[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 467054158 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002265] x2apic enabled [ 0.003005] Switched APIC routing to physical x2apic. [ 0.004007] kvm-guest: setup PV IPIs [ 0.006395] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008011] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009008] pid_max: default: 32768 minimum: 301 [ 0.010125] LSM: Security Framework initializing [ 0.011039] Yama: becoming mindful. [ 0.012027] SELinux: Initializing. [ 0.013054] *** VALIDATE selinux *** [ 0.020589] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024637] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025152] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026108] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027123] *** VALIDATE tmpfs *** [ 0.029285] *** VALIDATE proc *** [ 0.030185] *** VALIDATE cgroup *** [ 0.031008] *** VALIDATE cgroup2 *** [ 0.032179] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.033112] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.034006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.035033] Spectre V2 : User space: Vulnerable [ 0.036006] Speculative Store Bypass: Vulnerable [ 0.039158] debug: unmapping init [mem 0xffffffffa3259000-0xffffffffa3260fff] [ 0.041135] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042570] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043016] ... version: 2 [ 0.043884] ... bit width: 48 [ 0.044012] ... generic registers: 4 [ 0.044875] ... value mask: 0000ffffffffffff [ 0.045020] ... max period: 00007fffffffffff [ 0.046020] ... fixed-purpose events: 3 [ 0.046913] ... event mask: 000000070000000f [ 0.047328] rcu: Hierarchical SRCU implementation. [ 0.049703] smp: Bringing up secondary CPUs ... [ 0.050672] x86: Booting SMP configuration: [ 0.051029] .... node #0, CPUs: #1 #2 #3 [ 0.055020] smp: Brought up 1 node, 4 CPUs [ 0.057017] smpboot: Max logical packages: 1 [ 0.058027] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.141309] node 0 deferred pages initialised in 79ms [ 0.144096] devtmpfs: initialized [ 0.145271] x86/mm: Memory block size: 128MB [ 0.148926] gcov: version magic: 0x41383552 [ 0.152229] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.153071] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.154318] pinctrl core: initialized pinctrl subsystem [ 0.155156] [ 0.155487] ************************************************************* [ 0.157011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.159008] ** ** [ 0.160007] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.162008] ** ** [ 0.163008] ** This means that this kernel is built to expose internal ** [ 0.165010] ** IOMMU data structures, which may compromise security on ** [ 0.166007] ** your system. ** [ 0.168012] ** ** [ 0.169007] ** If you see this message and you are not debugging the ** [ 0.171008] ** kernel, report this immediately to your vendor! ** [ 0.172008] ** ** [ 0.173010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.175007] ************************************************************* [ 0.176524] NET: Registered protocol family 16 [ 0.177292] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.179045] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.181039] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.183367] cpuidle: using governor menu [ 0.186151] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.188447] PCI: Using configuration type 1 for base access [ 0.191121] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.200154] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.202033] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.206033] cryptd: max_cpu_qlen set to 1000 [ 0.210263] ACPI: Added _OSI(Module Device) [ 0.211014] ACPI: Added _OSI(Processor Device) [ 0.213013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.215013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.219072] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.224340] ACPI: Interpreter enabled [ 0.226049] ACPI: PM: (supports S0 S3 S4 S5) [ 0.228015] ACPI: Using IOAPIC for interrupt routing [ 0.229115] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.232387] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.243324] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.245043] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.248020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.251090] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.256266] acpiphp: Slot [2] registered [ 0.258063] acpiphp: Slot [5] registered [ 0.259072] acpiphp: Slot [6] registered [ 0.260041] acpiphp: Slot [3] registered [ 0.260987] acpiphp: Slot [4] registered [ 0.262052] acpiphp: Slot [7] registered [ 0.262960] acpiphp: Slot [8] registered [ 0.263058] acpiphp: Slot [9] registered [ 0.263867] acpiphp: Slot [10] registered [ 0.265082] acpiphp: Slot [11] registered [ 0.266099] acpiphp: Slot [12] registered [ 0.267072] acpiphp: Slot [13] registered [ 0.267934] acpiphp: Slot [14] registered [ 0.269112] acpiphp: Slot [15] registered [ 0.270056] acpiphp: Slot [16] registered [ 0.271082] acpiphp: Slot [17] registered [ 0.272021] acpiphp: Slot [18] registered [ 0.273055] acpiphp: Slot [19] registered [ 0.274052] acpiphp: Slot [20] registered [ 0.275047] acpiphp: Slot [21] registered [ 0.276020] acpiphp: Slot [22] registered [ 0.277091] acpiphp: Slot [23] registered [ 0.278042] acpiphp: Slot [24] registered [ 0.279048] acpiphp: Slot [25] registered [ 0.279924] acpiphp: Slot [26] registered [ 0.281107] acpiphp: Slot [27] registered [ 0.282020] acpiphp: Slot [28] registered [ 0.283092] acpiphp: Slot [29] registered [ 0.284095] acpiphp: Slot [30] registered [ 0.286099] acpiphp: Slot [31] registered [ 0.287057] PCI host bridge to bus 0000:00 [ 0.288026] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.290019] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.292021] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.294017] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.295015] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.297027] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.299164] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.300797] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.302976] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.309958] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.312553] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.315020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.317015] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.318013] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.320476] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.322547] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.324040] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.328318] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.331837] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.342729] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.347015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.353447] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.360984] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.365017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.373015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.382041] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.390019] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.398024] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.416037] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.426538] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.428324] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.430316] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.432287] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.434159] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.438106] iommu: Default domain type: Passthrough [ 0.439449] SCSI subsystem initialized [ 0.441122] ACPI: bus type USB registered [ 0.442084] usbcore: registered new interface driver usbfs [ 0.443065] usbcore: registered new interface driver hub [ 0.444061] usbcore: registered new device driver usb [ 0.446149] pps_core: LinuxPPS API ver. 1 registered [ 0.447010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.449050] PTP clock support registered [ 0.451034] EDAC MC: Ver: 3.0.0 [ 0.452099] PCI: Using ACPI for IRQ routing [ 0.453661] NetLabel: Initializing [ 0.454008] NetLabel: domain hash size = 128 [ 0.455007] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.457070] NetLabel: unlabeled traffic allowed by default [ 0.458113] vgaarb: loaded [ 0.459233] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.460008] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.465335] clocksource: Switched to clocksource kvm-clock [ 0.553682] VFS: Disk quotas dquot_6.6.0 [ 0.555067] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.556850] *** VALIDATE ramfs *** [ 0.557861] *** VALIDATE hugetlbfs *** [ 0.559247] pnp: PnP ACPI init [ 0.561402] pnp: PnP ACPI: found 6 devices [ 0.577645] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.580862] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.582806] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.584970] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.587275] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.589584] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.591918] NET: Registered protocol family 2 [ 0.593767] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.597797] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.601206] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.606376] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.609279] TCP: Hash tables configured (established 65536 bind 65536) [ 0.611955] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.615172] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.618154] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.620900] NET: Registered protocol family 1 [ 0.623369] RPC: Registered named UNIX socket transport module. [ 0.626072] RPC: Registered udp transport module. [ 0.628210] RPC: Registered tcp transport module. [ 0.630205] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.632838] NET: Registered protocol family 44 [ 0.635443] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.637851] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.640217] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.642632] PCI: CLS 0 bytes, default 64 [ 0.644457] Unpacking initramfs... [ 2.042366] debug: unmapping init [mem 0xffff96a7fcc64000-0xffff96a7fffcffff] [ 2.046687] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.049114] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.050860] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.540165] Initialise system trusted keyrings [ 2.541950] Key type blacklist registered [ 2.543896] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.552720] zbud: loaded [ 2.555217] *** VALIDATE nfs *** [ 2.556232] *** VALIDATE nfs4 *** [ 2.557516] pstore: using deflate compression [ 2.560603] Platform Keyring initialized [ 2.666793] NET: Registered protocol family 38 [ 2.668122] Key type asymmetric registered [ 2.669384] Asymmetric key parser 'x509' registered [ 2.670451] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.672502] io scheduler mq-deadline registered [ 2.674270] io scheduler kyber registered [ 2.676165] io scheduler bfq registered [ 2.678064] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.680982] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.683851] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.686803] ACPI: Power Button [PWRF] [ 2.692236] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.696770] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.703782] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.729650] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.757548] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.762263] Non-volatile memory driver v1.3 [ 2.763810] Linux agpgart interface v0.103 [ 2.791890] virtio_blk virtio1: [vda] 136552 512-byte logical blocks (69.9 MB/66.7 MiB) [ 2.794555] vda: detected capacity change from 0 to 69914624 [ 2.809439] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.812394] vdb: detected capacity change from 0 to 1073741824 [ 2.818420] libphy: Fixed MDIO Bus: probed [ 2.823845] usbcore: registered new interface driver usbserial_generic [ 2.826626] usbserial: USB Serial support registered for generic [ 2.828139] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.831052] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.832514] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.834601] mousedev: PS/2 mouse device common for all mice [ 2.837164] rtc_cmos 00:05: RTC can wake from S4 [ 2.839841] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.843264] rtc_cmos 00:05: registered as rtc0 [ 2.845558] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.846698] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.851783] intel_pstate: CPU model not supported [ 2.855906] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.860695] hid: raw HID events driver (C) Jiri Kosina [ 2.862155] usbcore: registered new interface driver usbhid [ 2.863624] usbhid: USB HID core driver [ 2.865351] drop_monitor: Initializing network drop monitor service [ 2.867230] Initializing XFRM netlink socket [ 2.868839] NET: Registered protocol family 10 [ 2.871267] Segment Routing with IPv6 [ 2.872378] NET: Registered protocol family 17 [ 2.873793] mpls_gso: MPLS GSO support [ 2.879149] RAS: Correctable Errors collector initialized. [ 2.881165] AVX version of gcm_enc/dec engaged. [ 2.882374] AES CTR mode by8 optimization enabled [ 2.952225] sched_clock: Marking stable (2952189116, 0)->(3715464478, -763275362) [ 2.955805] registered taskstats version 1 [ 2.957852] Loading compiled-in X.509 certificates [ 2.959946] zswap: loaded using pool lzo/zbud [ 2.984855] Key type big_key registered [ 2.996834] Key type encrypted registered [ 2.998563] ima: No TPM chip found, activating TPM-bypass! [ 3.000883] ima: Allocated hash algorithm: sha1 [ 3.002469] ima: No architecture policies found [ 3.004150] evm: Initialising EVM extended attributes: [ 3.005920] evm: security.selinux [ 3.007170] evm: security.ima [ 3.008243] evm: security.capability [ 3.009539] evm: HMAC attrs: 0x1 [ 3.011841] rtc_cmos 00:05: setting system clock to 2026-06-11 23:00:17 UTC (1781218817) [ 3.017402] debug: unmapping init [mem 0xffffffffa4203000-0xffffffffa43fffff] [ 3.020072] debug: unmapping init [mem 0xffffffffa2f82000-0xffffffffa3258fff] [ 3.033094] Write protecting the kernel read-only data: 28672k [ 3.036858] debug: unmapping init [mem 0xffffffffa1603000-0xffffffffa17fffff] [ 3.039328] debug: unmapping init [mem 0xffffffffa1f14000-0xffffffffa1ffffff] [ 3.071851] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.080374] systemd[1]: Detected virtualization kvm. [ 3.082184] systemd[1]: Detected architecture x86-64. [ 3.084118] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.111049] systemd[1]: No hostname configured. [ 3.112549] systemd[1]: Set hostname to . [ 3.114512] random: systemd: uninitialized urandom read (16 bytes read) [ 3.116957] systemd[1]: Initializing machine ID from random generator. [ 3.170057] random: ln: uninitialized urandom read (6 bytes read) [ 3.246089] random: systemd: uninitialized urandom read (16 bytes read) [ 3.248070] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.251470] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.255133] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.952974] device-mapper: uevent: version 1.0.3 [ 3.955947] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 6.243327] virtio_net virtio0 ens2: renamed from eth0 [ 6.401065] random: fast init done [ 6.568727] scsi host0: ata_piix [ 6.580899] scsi host1: ata_piix [ 6.585874] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 6.594693] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 11.532105] random: crng init done [ 11.533542] random: 7 urandom warning(s) missed due to ratelimiting [ 13.109473] dracut-initqueue[587]: RTNETLINK answers: File exists [ 13.828180] hrtimer: interrupt took 5252031 ns Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 15.232358] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 18.766756] printk: systemd: 26 output lines suppressed due to ratelimiting [ 19.624878] SELinux: Disabled at runtime. [ 19.750977] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 19.762108] systemd[1]: Detected virtualization kvm. [ 19.764562] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 21.465729] systemd[1]: initrd-switch-root.service: Succeeded. [ 21.469796] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 21.477873] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 21.483068] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 21.490818] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 21.499954] systemd[1]: Starting Journal Service... Starting Journal Service... [ 21.513158] systemd[1]: Activating swap /dev/disk/by-label/SWAP... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice User and Session Slice. [ 21.569888] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target rpc_pipefs.target. Starting Remount Root and Kernel File Systems... Starting Apply Kernel Variables... Mounting Huge Pages File System... [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting POSIX Message Queue File System... [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Kernel Socket. Mounting Kernel Debug File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 22.796655] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 23.499715] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 23.518425] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 24.118671] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 24.213755] EDAC sbridge: Ver: 1.1.2 [ 28.050461] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support (7s / no limit)[ 29.101848] NFS: Registering the id_resolver key type [ 29.106434] Key type id_resolver registered [ 29.107805] Key type id_legacy registered [** ] A start job is running for Configur…-only root support (7s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg235-client login: [ 105.103112] libcfs: loading out-of-tree module taints kernel. [ 105.379256] Key type ._llcrypt registered [ 105.388624] Key type .llcrypt registered [ 106.237893] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 106.250485] alg: No test for adler32 (adler32-zlib) [ 107.989393] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 109.308383] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 111.299533] Key type lgssc registered [ 113.585254] Lustre: Echo OBD driver; http://www.lustre.org/ [ 261.063160] Lustre: Mounted lustre-client [ 266.354084] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 280.596859] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing check_logdir /tmp/testlogs/ [ 286.694665] Lustre: lustre-OST0000-osc-ffff96a85171b000: disconnect after 24s idle [ 287.009780] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing yml_node [ 292.787661] Lustre: DEBUG MARKER: Client: 2.17.53.74 [ 295.765947] Lustre: DEBUG MARKER: MDS: 2.17.53.74 [ 299.404475] Lustre: DEBUG MARKER: OSS: 2.17.53.74 [ 301.743159] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Thu Jun 11 19:05:14 EDT 2026 [ 322.868077] Lustre: Unmounted lustre-client [ 345.019626] Key type lgssc unregistered [ 345.435480] LNet: 8394:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 345.443633] LNetError: 8394:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 345.464649] LNet: Removed LNI 192.168.202.35@tcp [ 346.575822] Key type .llcrypt unregistered [ 346.580159] Key type ._llcrypt unregistered [ 381.346273] Key type ._llcrypt registered [ 381.347303] Key type .llcrypt registered [ 381.483633] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 381.496320] alg: No test for adler32 (adler32-zlib) [ 382.565828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 382.638685] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 382.672575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 387.734295] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 388.102345] LNet: 9299:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 388.108214] LNetError: 9299:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 388.133956] LNet: Removed LNI 192.168.202.35@tcp [ 388.926590] Key type .llcrypt unregistered [ 388.932329] Key type ._llcrypt unregistered [ 398.498315] Key type ._llcrypt registered [ 398.510073] Key type .llcrypt registered [ 400.959346] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 19:06:52 (1781219212) [ 401.618743] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 401.670919] alg: No test for adler32 (adler32-zlib) [ 402.722347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 402.948264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 402.977946] LNet: 9711:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 403.031662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 403.128522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 403.190653] LNet: 9718:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 413.323399] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 19:07:05 (1781219225) [ 413.505233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 413.577100] LNet: 10289:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 413.658290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 413.973364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 425.017133] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 19:07:17 (1781219237) [ 425.413554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 425.485506] LNet: 10885:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 425.559061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 425.766183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 435.691983] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 19:07:28 (1781219248) [ 435.849573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 435.916223] LNet: 11464:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 435.961300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 436.091465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 447.243900] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 19:07:39 (1781219259) [ 447.352590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 447.411494] LNet: 12048:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 447.454594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 447.688092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 448.486307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 458.152944] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 19:07:50 (1781219270) [ 458.274171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 458.339108] LNet: 12640:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 458.394070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 458.589761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 459.397207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 459.988050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 468.789518] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 19:08:01 (1781219281) [ 469.011042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 469.108974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 469.244573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 479.260500] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 19:08:11 (1781219291) [ 479.362699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 479.398876] LNet: 13820:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 479.403336] LNet: 13820:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 479.431675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 479.519000] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 479.629593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 480.520862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 480.593774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 481.104551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 481.166466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 481.918649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 481.959274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 482.525912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 482.569756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 483.141442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 483.181838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 483.758433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 483.808730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 484.515973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 484.566163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 485.203246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 485.252714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 485.750667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 485.798780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 486.407710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 486.457954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 487.113491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 487.153514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 487.494874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 487.529962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 495.011436] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 19:08:28 (1781219308) [ 495.237306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 495.277914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 495.416599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 495.498902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 504.219992] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 19:08:36 (1781219316) [ 504.359489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 504.414519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 504.515643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 504.578446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 512.834319] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 19:08:45 (1781219325) [ 513.013182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 513.061323] LNet: 15678:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 513.065133] LNet: 15678:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 513.090956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 513.189924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 513.264132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 522.563253] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 19:08:54 (1781219334) [ 522.674583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 522.733466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 522.869637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 522.970229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 532.950462] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 19:09:05 (1781219345) [ 533.084130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 533.171159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 533.315988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 533.400908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 542.796688] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 19:09:15 (1781219355) [ 542.891119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 542.956251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 543.076130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 543.205987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 553.075199] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 19:09:25 (1781219365) [ 553.205928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 553.279584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 553.482912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 554.415403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 564.644870] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 19:09:36 (1781219376) [ 564.787372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 564.861696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 565.238255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 565.611499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 576.578499] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 19:09:48 (1781219388) [ 576.782730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 576.909993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 577.103592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 577.305213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 587.100691] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 19:09:59 (1781219399) [ 587.241526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 587.276574] LNet: 19857:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 587.284239] LNet: 19857:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 6 previous similar messages [ 587.340191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 588.365684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 588.821544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 599.383386] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 19:10:11 (1781219411) [ 599.503714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 599.570590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 599.689751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 599.775630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 609.326929] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 19:10:21 (1781219421) [ 609.527945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 609.644181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 609.829235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 609.898892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 619.137477] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 19:10:31 (1781219431) [ 619.280714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 619.351508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 619.483629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 628.475385] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 19:10:41 (1781219441) [ 628.734497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 628.813811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 628.954473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 637.794336] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 19:10:50 (1781219450) [ 637.918652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 637.991913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 638.091459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 638.855990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 648.475755] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 19:11:00 (1781219460) [ 648.607494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 648.685864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 648.879654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 648.937246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 657.484754] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 19:11:10 (1781219470) [ 657.591773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 657.690607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 657.838248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 657.925704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 667.388637] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 19:11:19 (1781219479) [ 667.620397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 667.756633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 667.912514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 668.033708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 677.687917] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 19:11:30 (1781219490) [ 678.078370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 678.172524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 678.334628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 678.412497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 688.026890] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 19:11:40 (1781219500) [ 688.169327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 688.199944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 688.275708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 688.404615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 688.428888] LNetError: 25841:0:(peer.c:2175:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 688.497582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 688.539542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 696.664818] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 19:11:49 (1781219509) [ 696.765163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 696.821988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 696.904896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.t55Ph2qf4E [ 696.972142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.t55Ph2qf4E [ 697.026632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.t55Ph2qf4E [ 705.274479] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 19:11:57 (1781219517) [ 705.412462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 705.464768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 705.563778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 705.656801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 705.714843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 714.214741] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 19:12:06 (1781219526) [ 714.427729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 714.501238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 714.626439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 719.198976] LNet: Message forwarding will be enabled in 2s [ 722.271177] LNet: Message forwarding enabled [ 722.858552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 722.897674] LNet: Message forwarding will stop in 2s to 6s [ 727.538916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 728.858934] LNet: Message forwarding disabled [ 730.764705] LNet: Message forwarding will be enabled in 2s [ 733.985330] LNet: Message forwarding enabled [ 740.189327] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 19:12:32 (1781219552) [ 740.337709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 740.372339] LNet: 28283:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 740.399580] LNet: 28283:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 13 previous similar messages [ 742.079714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 742.178886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 751.054515] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 19:12:43 (1781219563) [ 751.685635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 751.737899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 751.807209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 751.839674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 751.883889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 751.929262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 751.967235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 752.008807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 752.049497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 752.093596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 752.136881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 752.186247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 752.233816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 752.276838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 752.326520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 752.383767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 752.431828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 752.484840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 752.543562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 752.602636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 752.675064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 752.710964] LNetError: 28916:0:(router.c:624:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 760.967985] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 19:12:53 (1781219573) [ 761.509989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 761.597907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 761.716818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 770.027211] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 19:13:02 (1781219582) [ 772.476212] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg235-server [ 774.511944] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 19:13:07 (1781219587) [ 774.988944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 775.050987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 775.134249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 775.175168] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 777.474322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.202.[36-40]@tcp [ 777.528328] LNet: 30635:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 777.611784] LNetError: 30424:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.40@tcp is being used as a gateway but routing feature is not turned on [ 777.626966] LNetError: 30424:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp1 through 192.168.202.40@tcp has gone from up to down [ 778.599818] LNetError: 30424:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.36@tcp is being used as a gateway but routing feature is not turned on [ 778.620301] LNetError: 30424:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 779.626526] LNetError: 30424:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.39@tcp is being used as a gateway but routing feature is not turned on [ 779.650931] LNetError: 30424:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 781.677593] LNetError: 30424:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.37@tcp is being used as a gateway but routing feature is not turned on [ 781.699360] LNetError: 30424:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 785.324589] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 19:13:18 (1781219598) [ 785.790454] LNetError: 30424:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.39@tcp is being used as a gateway but routing feature is not turned on [ 785.805780] LNetError: 30424:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 787.639192] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg235-server [ 789.500798] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 19:13:22 (1781219602) [ 789.669522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 789.722932] LNetError: 31566:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 789.773620] LNet: Removed LNI 192.168.202.35@tcp [ 789.833328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 789.936359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 789.962331] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 790.826879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.202.[36-41/2]@tcp [ 790.856141] LNet: 31689:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 790.863540] LNet: 31689:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 790.903412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.202.[36-41/2]@tcp [ 790.912595] LNetError: 31576:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp103 through 192.168.202.38@tcp has gone from up to down [ 790.947890] LNetError: 31576:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 798.458841] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 19:13:31 (1781219611) [ 798.795867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 798.824897] LNetError: 32283:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 798.848683] LNet: Removed LNI 192.168.202.35@tcp [ 798.870953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 798.993835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 799.137539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 799.190146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 799.264685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 799.338424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 799.416317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 799.493471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 799.605876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 799.694853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 799.772672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 799.869620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 799.906461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 799.977933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 800.115462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 800.156911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 800.243906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 800.331772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 800.364450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 800.453180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 800.634477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 800.674396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 800.771941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 800.939894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 801.005279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 801.072838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 801.193200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 801.251696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 801.392433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 801.513608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 801.567660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 801.657549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 801.762556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 801.815832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 801.910486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 810.156472] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 19:13:43 (1781219623) [ 812.739764] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg235-server [ 815.191315] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 19:13:47 (1781219627) [ 817.136448] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg235-server [ 819.397323] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 19:13:52 (1781219632) [ 820.499878] Key type .llcrypt unregistered [ 820.505379] Key type ._llcrypt unregistered [ 827.370888] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 827.550974] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 827.553568] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 827.566737] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 829.355566] Key type ._llcrypt registered [ 829.357138] Key type .llcrypt registered [ 829.685596] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 829.718419] alg: No test for adler32 (adler32-zlib) [ 830.690381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 830.787600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 830.826583] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 831.882704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 831.914034] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 834.743348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 834.774910] LNet: Removed LNI 10.1.2.3@tcp [ 834.970087] LNet: 34399:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 834.997612] LNetError: 34399:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 835.046270] LNet: Removed LNI 192.168.202.35@tcp [ 836.024992] Key type .llcrypt unregistered [ 836.029087] Key type ._llcrypt unregistered [ 851.374671] Key type ._llcrypt registered [ 851.387244] Key type .llcrypt registered [ 853.431721] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 19:14:25 (1781219665) [ 854.396490] Key type .llcrypt unregistered [ 854.404265] Key type ._llcrypt unregistered [ 862.243606] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 862.494044] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 862.531576] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 862.554756] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 864.427635] Key type ._llcrypt registered [ 864.429424] Key type .llcrypt registered [ 864.813541] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 864.836978] alg: No test for adler32 (adler32-zlib) [ 865.802046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 865.862749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 865.884494] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 867.033488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 867.076649] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 870.682938] LNet: 35882:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 870.690225] LNetError: 35882:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 870.696164] LNet: Removed LNI 192.168.202.35@tcp [ 870.716503] LNet: Removed LNI 10.1.2.3@tcp [ 871.574597] Key type .llcrypt unregistered [ 871.578109] Key type ._llcrypt unregistered [ 886.503615] Key type ._llcrypt registered [ 886.507692] Key type .llcrypt registered [ 888.222639] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 19:15:00 (1781219700) [ 889.017540] Key type .llcrypt unregistered [ 889.018712] Key type ._llcrypt unregistered [ 896.448645] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 896.761619] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 896.776502] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 896.796059] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 899.878334] Key type ._llcrypt registered [ 899.882766] Key type .llcrypt registered [ 900.185411] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 900.207717] alg: No test for adler32 (adler32-zlib) [ 901.160358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 901.252954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 901.291265] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 904.092843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 904.139879] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 907.930731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 907.965624] LNet: Removed LNI 10.1.2.3@tcp [ 907.991850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 908.046802] LNet: Removed LNI 10.1.2.31@tcp [ 908.082846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 908.135106] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 909.530579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 909.583871] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 911.373125] LNet: 37805:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 911.401673] LNetError: 37805:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 911.423147] LNet: Removed LNI 10.1.2.3@tcp [ 911.449509] LNet: Removed LNI 10.1.2.31@tcp [ 912.569731] Key type .llcrypt unregistered [ 912.574858] Key type ._llcrypt unregistered [ 927.219298] Key type ._llcrypt registered [ 927.220377] Key type .llcrypt registered [ 928.548743] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 19:15:41 (1781219741) [ 929.221296] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 929.244153] alg: No test for adler32 (adler32-zlib) [ 930.330589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 930.395289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 930.413751] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 931.835351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 931.871230] LNet: 38760:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 931.876235] LNetError: 38760:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 931.886773] LNet: Removed LNI 192.168.202.35@tcp [ 931.921359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 932.024499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.202.35@tcp [ 932.083330] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 933.443461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.202.35@tcp [ 933.536208] LNet: Removed LNI 192.168.202.35@tcp [ 933.659471] LNet: 38895:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 934.655603] Key type .llcrypt unregistered [ 934.663766] Key type ._llcrypt unregistered [ 950.627526] Key type ._llcrypt registered [ 950.629258] Key type .llcrypt registered [ 953.084889] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 19:16:04 (1781219764) [ 956.171395] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg235-server [ 958.843480] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 19:16:11 (1781219771) [ 960.266110] Key type .llcrypt unregistered [ 960.283171] Key type ._llcrypt unregistered [ 970.162267] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 970.351376] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 970.362678] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 970.398807] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 973.581532] Key type ._llcrypt registered [ 973.584432] Key type .llcrypt registered [ 973.936690] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 973.963455] alg: No test for adler32 (adler32-zlib) [ 974.973603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 975.072416] LNet: Added LNI 192.168.202.35@tcp1 [8/256/0/180] [ 976.194784] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 979.393719] LNet: 40734:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 979.405763] LNetError: 40734:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 979.415480] LNet: Removed LNI 192.168.202.35@tcp1 [ 979.434840] LNet: Removed LNI 10.1.2.3@tcp2 [ 980.511549] Key type .llcrypt unregistered [ 980.517528] Key type ._llcrypt unregistered [ 996.257885] Key type ._llcrypt registered [ 996.262706] Key type .llcrypt registered [ 998.218738] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 19:16:50 (1781219810) [ 998.570801] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 998.590416] alg: No test for adler32 (adler32-zlib) [ 999.595136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 999.658923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 999.687523] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1000.928337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1000.970941] LNet: 41704:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1000.981252] LNetError: 41704:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1000.992875] LNet: Removed LNI 192.168.202.35@tcp [ 1001.009718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1001.083236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 1001.136596] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1011.621250] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 19:17:03 (1781219823) [ 1011.800696] LNet: 42287:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1011.815433] LNetError: 42287:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1011.839305] LNet: Removed LNI 192.168.202.35@tcp [ 1013.007644] Key type .llcrypt unregistered [ 1013.012131] Key type ._llcrypt unregistered [ 1020.396446] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1020.707751] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1020.723658] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1020.741575] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1023.793374] Key type ._llcrypt registered [ 1023.796630] Key type .llcrypt registered [ 1024.175039] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1024.233101] alg: No test for adler32 (adler32-zlib) [ 1025.379602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1025.492869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1025.542729] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1027.027755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1027.071827] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1030.351798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1030.414500] LNet: 43041:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1030.422648] LNetError: 43041:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1030.428536] LNet: Removed LNI 192.168.202.35@tcp [ 1030.438633] LNet: Removed LNI 10.1.2.3@tcp [ 1030.472472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1030.642467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 1030.712841] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1030.724229] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1031.760654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1031.815609] LNet: 43068:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1031.827502] LNetError: 43068:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1031.837167] LNetError: 43068:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1031.848645] LNet: Removed LNI 192.168.202.35@tcp [ 1031.858702] LNet: Removed LNI 10.1.2.3@tcp [ 1031.899440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1032.056296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 1032.153075] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1032.160753] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1033.061879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1033.078944] LNet: Added LNI 192.168.202.35@tcp2 [8/256/0/180] [ 1034.720267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1034.750640] LNet: 43231:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1034.761573] LNetError: 43231:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1034.778633] LNetError: 43231:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1034.791986] LNet: Removed LNI 192.168.202.35@tcp [ 1034.793953] LNet: Removed LNI 10.1.2.3@tcp [ 1034.805400] LNet: Removed LNI 192.168.202.35@tcp2 [ 1034.854349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1034.969270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 1034.991085] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1035.006561] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1035.027595] LNet: Added LNI 192.168.202.35@tcp2 [8/256/0/180] [ 1045.420749] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 19:17:37 (1781219857) [ 1045.670464] LNet: 43824:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1045.683476] LNetError: 43824:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1045.690652] LNetError: 43824:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1045.706601] LNet: Removed LNI 192.168.202.35@tcp [ 1045.709615] LNet: Removed LNI 10.1.2.3@tcp [ 1045.719493] LNet: Removed LNI 192.168.202.35@tcp2 [ 1046.521154] Key type .llcrypt unregistered [ 1046.525353] Key type ._llcrypt unregistered [ 1054.646698] Key type ._llcrypt registered [ 1054.656163] Key type .llcrypt registered [ 1055.118304] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1055.151693] alg: No test for adler32 (adler32-zlib) [ 1056.191848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1056.304970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1056.342721] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1057.640500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1057.711988] LNet: 44224:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1057.724324] LNetError: 44224:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1057.760699] LNet: Removed LNI 192.168.202.35@tcp [ 1057.826516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1057.974625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 1058.017042] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1059.284299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1059.340811] LNet: 44254:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1059.352725] LNetError: 44254:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1059.379274] LNet: Removed LNI 192.168.202.35@tcp [ 1059.424220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1059.545671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 1059.596538] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1060.790675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1060.844724] LNet: 44284:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1060.850867] LNetError: 44284:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1060.870189] LNet: Removed LNI 192.168.202.35@tcp [ 1060.916913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1061.046388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 1061.095684] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1062.024986] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1062.206744] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1062.218948] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1062.239051] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1065.102824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1065.122075] LNet: 44363:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1065.130193] LNetError: 44363:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1065.164904] LNet: Removed LNI 192.168.202.35@tcp [ 1065.192105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1065.323897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 1065.367290] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1074.430671] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 19:18:07 (1781219887) [ 1074.804100] LNet: 44961:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1074.820254] LNetError: 44961:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1074.835132] LNet: Removed LNI 192.168.202.35@tcp [ 1075.733088] Key type .llcrypt unregistered [ 1075.734761] Key type ._llcrypt unregistered [ 1084.051365] Key type ._llcrypt registered [ 1084.071310] Key type .llcrypt registered [ 1084.401799] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1084.431216] alg: No test for adler32 (adler32-zlib) [ 1085.394980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1085.447686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1085.470822] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1086.604992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1086.639870] LNet: 45360:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1086.675942] LNetError: 45360:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1086.716604] LNet: Removed LNI 192.168.202.35@tcp [ 1086.775799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1086.938211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 1086.983609] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1097.613965] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 19:18:29 (1781219909) [ 1097.754809] LNet: 45945:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1097.765906] LNetError: 45945:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1097.795761] LNet: Removed LNI 192.168.202.35@tcp [ 1098.781235] Key type .llcrypt unregistered [ 1098.792281] Key type ._llcrypt unregistered [ 1106.924345] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1107.340081] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1107.352342] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1107.368680] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1110.395310] Key type ._llcrypt registered [ 1110.400235] Key type .llcrypt registered [ 1110.709109] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1110.738615] alg: No test for adler32 (adler32-zlib) [ 1111.870723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1111.998697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1112.084047] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1114.217919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1114.264087] LNet: 46407:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1114.287955] LNetError: 46407:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1114.318985] LNet: Removed LNI 192.168.202.35@tcp [ 1114.347792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1114.505670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 1114.568518] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1127.114917] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 19:18:59 (1781219939) [ 1127.235909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1127.292575] LNet: 47002:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1127.310691] LNetError: 47002:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1127.356291] LNet: Removed LNI 192.168.202.35@tcp [ 1127.388612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1127.660275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 1138.099645] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 19:19:10 (1781219950) [ 1138.317410] LNet: 47583:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1139.268681] Key type .llcrypt unregistered [ 1139.272901] Key type ._llcrypt unregistered [ 1148.436163] Key type ._llcrypt registered [ 1148.440594] Key type .llcrypt registered [ 1148.942896] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1149.001435] alg: No test for adler32 (adler32-zlib) [ 1150.033973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1150.112377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1150.136893] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1151.524437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1151.565548] LNet: 47991:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1151.583731] LNetError: 47991:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1151.603162] LNet: Removed LNI 192.168.202.35@tcp [ 1151.637488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1151.827898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 1151.900695] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1162.391183] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 19:19:34 (1781219974) [ 1162.473705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1162.516800] LNet: 48579:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1162.524545] LNetError: 48579:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1162.550987] LNet: Removed LNI 192.168.202.35@tcp [ 1162.617192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1162.758982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 1171.988424] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 19:19:44 (1781219984) [ 1174.235811] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 1176.670832] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 19:19:49 (1781219989) [ 1178.912189] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 1181.449587] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 19:19:53 (1781219993) [ 1183.452983] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 1185.307799] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 19:19:58 (1781219998) [ 1187.344400] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1189.699559] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 19:20:02 (1781220002) [ 1191.858532] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1194.106465] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 19:20:06 (1781220006) [ 1194.272345] LNet: 50630:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1195.364724] Key type .llcrypt unregistered [ 1195.367114] Key type ._llcrypt unregistered [ 1203.066921] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1203.143915] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1203.164391] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1203.171068] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1206.071709] Key type ._llcrypt registered [ 1206.075809] Key type .llcrypt registered [ 1206.360851] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1206.401316] alg: No test for adler32 (adler32-zlib) [ 1207.443357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1207.618556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1207.653330] LNet: Added LNI 192.168.202.35@tcp [8/512/0/180] [ 1207.659918] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1207.663832] LNet: 50961:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1207.900596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1207.941969] LNet: 50980:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1207.952436] LNetError: 50980:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1207.966589] LNet: Removed LNI 192.168.202.35@tcp [ 1207.970243] LNetError: 50980:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1207.993194] LNet: Removed LNI 10.1.2.3@tcp [ 1208.053980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1208.187947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1208.229423] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1208.382856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1208.469910] LNet: 51004:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1208.482644] LNetError: 51004:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1208.508991] LNet: Removed LNI 192.168.202.35@tcp [ 1208.587282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1208.666877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1208.712324] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1208.722805] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1208.727780] LNet: 51012:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1208.820966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1208.839263] LNet: 51030:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1208.843916] LNetError: 51030:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1208.848340] LNet: Removed LNI 192.168.202.35@tcp [ 1208.851304] LNetError: 51030:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1208.875384] LNet: Removed LNI 10.1.2.3@tcp [ 1208.917320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1209.018647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1209.069823] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1209.075867] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1209.081456] LNet: 51038:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1209.355405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1209.381517] LNet: 51064:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1209.387750] LNetError: 51064:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1209.392702] LNet: Removed LNI 192.168.202.35@tcp [ 1209.396392] LNetError: 51064:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1209.413729] LNet: Removed LNI 10.1.2.3@tcp [ 1209.445789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1209.537333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1209.566544] LNet: Added LNI 192.168.202.35@tcp [8/256/0/182] [ 1209.572610] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1209.702283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1209.748400] LNet: 51092:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1209.752949] LNetError: 51092:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1209.757777] LNet: Removed LNI 192.168.202.35@tcp [ 1209.760642] LNetError: 51092:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1209.771465] LNet: Removed LNI 10.1.2.3@tcp [ 1209.792382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1209.860847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1209.881687] LNet: Added LNI 192.168.202.35@tcp [8/256/0/182] [ 1209.888609] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1209.891613] LNet: 51100:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1209.970860] LNet: Removed LNI 192.168.202.35@tcp [ 1209.987317] LNet: Removed LNI 10.1.2.3@tcp [ 1210.714738] Key type .llcrypt unregistered [ 1210.717266] Key type ._llcrypt unregistered [ 1225.810255] Key type ._llcrypt registered [ 1225.812484] Key type .llcrypt registered [ 1227.766279] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 19:20:40 (1781220040) [ 1228.151395] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1228.166530] alg: No test for adler32 (adler32-zlib) [ 1229.151953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1229.202487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1229.225501] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1230.230775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.202.35 [ 1230.263469] LNet: 52077:0:(config.c:1627:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1230.282880] LNet: Removed LNI 192.168.202.35@tcp [ 1238.270499] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 19:20:51 (1781220051) [ 1238.526262] LNet: 52648:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1238.532460] LNetError: 52648:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1238.570307] LNet: Removed LNI 192.168.202.35@tcp [ 1239.382425] Key type .llcrypt unregistered [ 1239.390754] Key type ._llcrypt unregistered [ 1246.638504] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1246.971528] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1249.997236] Key type ._llcrypt registered [ 1250.002816] Key type .llcrypt registered [ 1250.283563] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1250.306672] alg: No test for adler32 (adler32-zlib) [ 1252.055565] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1261.517890] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 19:21:14 (1781220074) [ 1261.895178] LNet: 53562:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1261.922067] LNetError: 53562:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1261.949615] LNet: Removed LNI 10.1.2.3@tcp [ 1262.883810] Key type .llcrypt unregistered [ 1262.886403] Key type ._llcrypt unregistered [ 1271.340538] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1271.701833] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1271.705371] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1271.711463] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1274.730818] Key type ._llcrypt registered [ 1274.732614] Key type .llcrypt registered [ 1275.047207] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1275.059140] alg: No test for adler32 (adler32-zlib) [ 1276.129604] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1284.420143] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 19:21:37 (1781220097) [ 1286.204241] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1288.396677] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 19:21:41 (1781220101) [ 1288.527692] LNet: 54741:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1288.537904] LNetError: 54741:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1288.564560] LNet: Removed LNI 10.1.2.3@tcp [ 1289.488120] Key type .llcrypt unregistered [ 1289.493484] Key type ._llcrypt unregistered [ 1296.625986] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1297.040988] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1297.047072] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1297.058583] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1298.988520] Key type ._llcrypt registered [ 1298.990680] Key type .llcrypt registered [ 1299.275971] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1299.295513] alg: No test for adler32 (adler32-zlib) [ 1300.385987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1300.471756] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1312.534048] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 19:22:05 (1781220125) [ 1313.271160] LNet: 55972:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1313.280230] LNetError: 55972:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1313.293058] LNet: Removed LNI 10.1.2.3@tcp [ 1313.911745] Key type .llcrypt unregistered [ 1313.913319] Key type ._llcrypt unregistered [ 1320.807329] Key type ._llcrypt registered [ 1320.815421] Key type .llcrypt registered [ 1321.058431] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1321.087825] alg: No test for adler32 (adler32-zlib) [ 1322.146341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1322.181011] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1331.287578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.135@tcp [ 1332.447894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e local_interrupt -r 1 [ 1332.555847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e local_interrupt -r 1 [ 1332.625918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.135@tcp [ 1332.671746] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 1333.007732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1333.042563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1333.078629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1333.132659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1333.190683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1333.259205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1333.315347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1333.380721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1333.431706] LNet: 56411:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1333.467950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1333.471428] LNet: 1 local NIs in recovery (showing 1): 192.168.202.35@tcp [ 1333.474649] LNet: There was an unexpected network error while writing to 192.168.202.35: rc = -22 [ 1333.526578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1333.580721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1333.653752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1333.708913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1333.763809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1333.817580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1334.649557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e local_dropped -r 1 [ 1334.731225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e local_dropped -r 1 [ 1334.850903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.135@tcp [ 1334.921401] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 1335.301331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1335.372760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1335.409254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1335.434708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1335.456451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1335.482431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1335.525391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1335.598645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1335.636625] LNet: 56520:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1335.666434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1335.732740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1335.790660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1335.877636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1335.946213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1335.995516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1336.048904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1336.819572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e local_aborted -r 1 [ 1336.922841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e local_aborted -r 1 [ 1337.032656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.135@tcp [ 1337.092787] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 1337.401485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1337.460321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1337.520262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1337.589582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1337.642391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1337.697713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1337.770880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1337.816598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1337.853711] LNet: 56630:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1337.889784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1337.973896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1338.071940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1338.165925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1338.254886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1338.314408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1338.364784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1338.826076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e local_no_route -r 1 [ 1338.859550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e local_no_route -r 1 [ 1338.893642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.135@tcp [ 1339.194316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1339.225377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1339.259397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1339.299698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1339.330791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1339.364220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1339.418418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1339.469483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1339.517398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1339.547882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1339.577185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1339.622893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1339.667534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1339.698811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1339.724983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1340.223769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e local_timeout -r 1 [ 1340.292664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e local_timeout -r 1 [ 1340.342564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.135@tcp [ 1340.631644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1340.670538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1340.713577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1340.761446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1340.806331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1340.857602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1340.927964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1340.964278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1340.985217] LNet: 56849:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1340.995540] LNet: 56849:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1341.011441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1341.059606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1341.108060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1341.174451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1341.218738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1341.277972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1341.340714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1341.914603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e local_error -r 1 [ 1341.975427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e local_error -r 1 [ 1342.020303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.135@tcp [ 1342.044583] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 1342.047802] LNet: Skipped 3 previous similar messages [ 1342.301883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1342.323117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1342.363795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1342.408351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1342.436867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1342.479608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1342.512636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1342.551194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1342.596908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1342.636955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1342.698807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1342.722155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1342.761087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1342.848882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1342.911116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1343.263441] LNet: 56999:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1343.277514] LNetError: 56999:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1344.294153] LNet: Removed LNI 192.168.202.35@tcp [ 1344.771730] Key type .llcrypt unregistered [ 1344.775450] Key type ._llcrypt unregistered [ 1365.398510] Key type ._llcrypt registered [ 1365.402556] Key type .llcrypt registered [ 1367.290681] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 19:22:59 (1781220179) [ 1368.352336] Key type .llcrypt unregistered [ 1368.355235] Key type ._llcrypt unregistered [ 1376.377751] Key type ._llcrypt registered [ 1376.381380] Key type .llcrypt registered [ 1376.772672] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1376.812431] alg: No test for adler32 (adler32-zlib) [ 1377.886190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1377.943517] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1386.649798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.135@tcp [ 1398.609170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1398.701266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1398.769753] LNet: Added LNI 192.168.202.35@tcp1 [8/256/0/180] [ 1400.256666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e local_interrupt -r 1 [ 1400.307569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp1 -m GET -e local_interrupt -r 1 [ 1400.345521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e local_interrupt -r 1 [ 1400.389745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp1 -m GET -e local_interrupt -r 1 [ 1400.429085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp -m GET -e local_interrupt -r 1 [ 1400.471420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp1 -m GET -e local_interrupt -r 1 [ 1400.518609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp -m GET -e local_interrupt -r 1 [ 1400.559696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp1 -m GET -e local_interrupt -r 1 [ 1400.595064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.135@tcp [ 1400.612139] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 1400.617582] LNet: 1 local NIs in recovery (showing 1): 192.168.202.35@tcp [ 1400.929872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1400.985729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1401.037644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1401.104370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1401.184594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1401.248473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1401.301532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1401.336661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1401.369481] LNet: 58470:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1401.401543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1401.458740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1401.508870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1401.564552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1401.602662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1401.636021] LNet: There was an unexpected network error while writing to 192.168.202.35: rc = -22 [ 1401.657172] LNet: Skipped 3 previous similar messages [ 1401.664963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1401.721725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1402.271509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e local_dropped -r 1 [ 1402.325566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp1 -m GET -e local_dropped -r 1 [ 1402.379864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e local_dropped -r 1 [ 1402.432718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp1 -m GET -e local_dropped -r 1 [ 1402.513134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp -m GET -e local_dropped -r 1 [ 1402.573392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp1 -m GET -e local_dropped -r 1 [ 1402.612913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp -m GET -e local_dropped -r 1 [ 1402.690711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp1 -m GET -e local_dropped -r 1 [ 1402.746633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.135@tcp [ 1402.779120] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 1403.131911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1403.158185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1403.180817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1403.205815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1403.249482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1403.289856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1403.318702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1403.349984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1403.363458] LNet: 58592:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1403.381595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1403.409449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1403.434803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1403.464465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1403.499681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1403.535931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1403.555661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1404.038787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e local_aborted -r 1 [ 1404.079435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp1 -m GET -e local_aborted -r 1 [ 1404.118363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e local_aborted -r 1 [ 1404.145439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp1 -m GET -e local_aborted -r 1 [ 1404.167659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp -m GET -e local_aborted -r 1 [ 1404.210518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp1 -m GET -e local_aborted -r 1 [ 1404.306690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp -m GET -e local_aborted -r 1 [ 1404.371542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp1 -m GET -e local_aborted -r 1 [ 1404.400236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.135@tcp [ 1404.607270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1404.635126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1404.661711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1404.686632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1404.718842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1404.754431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1404.794740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1404.830755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1404.855155] LNet: 58713:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1404.879774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1404.999920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1405.080267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1405.154727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1405.202933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1405.246531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1405.283824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1405.851556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e local_no_route -r 1 [ 1405.880886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp1 -m GET -e local_no_route -r 1 [ 1405.913431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e local_no_route -r 1 [ 1405.945510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp1 -m GET -e local_no_route -r 1 [ 1405.982453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp -m GET -e local_no_route -r 1 [ 1406.017593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp1 -m GET -e local_no_route -r 1 [ 1406.055651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp -m GET -e local_no_route -r 1 [ 1406.098374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp1 -m GET -e local_no_route -r 1 [ 1406.143604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.135@tcp [ 1406.164766] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 1406.172822] LNet: Skipped 5 previous similar messages [ 1406.427363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1406.479318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1406.527923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1406.584729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1406.642689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1406.683490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1406.743889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1406.809095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1406.850364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1406.911443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1406.981173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1407.012386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1407.045642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1407.072892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1407.102811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1407.529720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e local_timeout -r 1 [ 1407.554854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp1 -m GET -e local_timeout -r 1 [ 1407.590671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e local_timeout -r 1 [ 1407.615500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp1 -m GET -e local_timeout -r 1 [ 1407.653539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp -m GET -e local_timeout -r 1 [ 1407.679586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp1 -m GET -e local_timeout -r 1 [ 1407.716962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp -m GET -e local_timeout -r 1 [ 1407.752573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp1 -m GET -e local_timeout -r 1 [ 1407.780828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.135@tcp [ 1408.018481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1408.047122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1408.089152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1408.164792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1408.263298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1408.343232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1408.399899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1408.483291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1408.494370] LNet: 58956:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1408.498351] LNet: 58956:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1408.526187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1408.551223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1408.607258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1408.633556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1408.667929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1408.700387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1408.740687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1409.233422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e local_error -r 1 [ 1409.281328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp1 -m GET -e local_error -r 1 [ 1409.337585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e local_error -r 1 [ 1409.400179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp1 -m GET -e local_error -r 1 [ 1409.432989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp -m GET -e local_error -r 1 [ 1409.477932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp1 -m GET -e local_error -r 1 [ 1409.548622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp -m GET -e local_error -r 1 [ 1409.581310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp1 -m GET -e local_error -r 1 [ 1409.619911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.135@tcp [ 1409.858142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1409.896813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1409.931649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1409.958623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1409.994508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1410.029856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1410.056523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1410.074587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1410.105855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1410.136509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1410.162214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1410.185749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1410.200881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1410.224217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1410.256396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1416.630906] LNet: 59153:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1416.637610] LNetError: 59153:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1417.695289] LNet: Removed LNI 192.168.202.35@tcp [ 1417.705817] LNetError: 59153:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1417.732347] LNet: Removed LNI 192.168.202.35@tcp1 [ 1418.592816] Key type .llcrypt unregistered [ 1418.594542] Key type ._llcrypt unregistered [ 1441.731730] Key type ._llcrypt registered [ 1441.739734] Key type .llcrypt registered [ 1443.293876] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 19:24:15 (1781220255) [ 1444.365562] Key type .llcrypt unregistered [ 1444.369716] Key type ._llcrypt unregistered [ 1451.781801] Key type ._llcrypt registered [ 1451.783647] Key type .llcrypt registered [ 1452.029278] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1452.058430] alg: No test for adler32 (adler32-zlib) [ 1453.003963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1453.031935] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1461.338276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.135@tcp [ 1462.237747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e remote_dropped -r 1 [ 1462.298502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e remote_dropped -r 1 [ 1462.355546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.135@tcp [ 1462.377600] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 1462.693713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1462.725089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1462.746485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1462.777649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1462.812423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1462.855562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1462.896508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1462.946522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1462.974870] LNet: 60406:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1463.001642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1463.051884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1463.093495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1463.130461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1463.170782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1463.206576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1463.235547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1463.733141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e remote_error -r 1 [ 1463.797286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e remote_error -r 1 [ 1463.859031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.135@tcp [ 1463.885256] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 1464.192180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1464.229899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1464.279442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1464.313732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1464.357961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1464.409236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1464.439102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1464.474387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1464.497411] LNet: 60516:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1464.551746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1464.612716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1464.661187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1464.694320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1464.749524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1464.801130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1464.843251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1465.414570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e remote_timeout -r 1 [ 1465.479660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e remote_timeout -r 1 [ 1465.563739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.135@tcp [ 1465.574581] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 1465.856703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1465.899258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1465.925733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1465.958121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1466.003546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1466.031212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1466.092569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1466.123339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1466.151525] LNet: 60625:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1466.186878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1466.234197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1466.269236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1466.337600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1466.397785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1466.424181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1466.447740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1466.739310] LNet: 60666:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1466.746694] LNetError: 60666:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1466.760172] LNet: Removed LNI 192.168.202.35@tcp [ 1467.095573] Key type .llcrypt unregistered [ 1467.099416] Key type ._llcrypt unregistered [ 1487.452452] Key type ._llcrypt registered [ 1487.455592] Key type .llcrypt registered [ 1488.771214] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 19:25:01 (1781220301) [ 1490.001674] Key type .llcrypt unregistered [ 1490.003783] Key type ._llcrypt unregistered [ 1496.261658] Key type ._llcrypt registered [ 1496.264419] Key type .llcrypt registered [ 1496.505559] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1496.518825] alg: No test for adler32 (adler32-zlib) [ 1497.474631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1497.500507] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1507.546158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.135@tcp [ 1519.420872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1519.508515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1519.546964] LNet: Added LNI 192.168.202.35@tcp1 [8/256/0/180] [ 1521.257474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e remote_dropped -r 1 [ 1521.324292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp1 -m GET -e remote_dropped -r 1 [ 1521.366295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e remote_dropped -r 1 [ 1521.411749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp1 -m GET -e remote_dropped -r 1 [ 1521.475173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp -m GET -e remote_dropped -r 1 [ 1521.554940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp1 -m GET -e remote_dropped -r 1 [ 1521.623370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp -m GET -e remote_dropped -r 1 [ 1521.696947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp1 -m GET -e remote_dropped -r 1 [ 1521.770478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.135@tcp [ 1521.792711] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 1521.798732] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.135@tcp [ 1522.084770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1522.110615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1522.146572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1522.218386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1522.274727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1522.309661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1522.363836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1522.420629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1522.472948] LNet: 62136:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1522.497336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1522.522867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1522.555986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1522.582691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1522.617994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1522.667995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1522.726281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1522.754720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1523.393843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e remote_error -r 1 [ 1523.454853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp1 -m GET -e remote_error -r 1 [ 1523.539437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e remote_error -r 1 [ 1523.596401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp1 -m GET -e remote_error -r 1 [ 1523.650254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp -m GET -e remote_error -r 1 [ 1523.699108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp1 -m GET -e remote_error -r 1 [ 1523.750866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp -m GET -e remote_error -r 1 [ 1523.816424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp1 -m GET -e remote_error -r 1 [ 1523.919405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.135@tcp [ 1523.952206] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 1523.960282] LNet: Skipped 2 previous similar messages [ 1524.176266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1524.201208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1524.231804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1524.289579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1524.327055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1524.358976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1524.387488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1524.419298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1524.434334] LNet: 62259:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1524.452373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1524.481081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1524.514388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1524.547857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1524.587382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1524.620238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1524.651512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1524.686317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1525.245196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e remote_timeout -r 1 [ 1525.280551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp1 -m GET -e remote_timeout -r 1 [ 1525.336853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e remote_timeout -r 1 [ 1525.402311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp1 -m GET -e remote_timeout -r 1 [ 1525.458162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp -m GET -e remote_timeout -r 1 [ 1525.505815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp1 -m GET -e remote_timeout -r 1 [ 1525.561761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp -m GET -e remote_timeout -r 1 [ 1525.611943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp1 -m GET -e remote_timeout -r 1 [ 1525.652913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.135@tcp [ 1525.681235] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 1525.899672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1525.931917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1525.967933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1526.016399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1526.062889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1526.089376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1526.115611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1526.140849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1526.162603] LNet: 62382:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1526.192698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1526.232173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1526.271237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1526.304192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1526.350864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1526.393987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1526.427480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1526.491050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1531.664856] LNet: 62460:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1531.668621] LNetError: 62460:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1531.671527] LNet: Removed LNI 192.168.202.35@tcp [ 1531.683936] LNet: Removed LNI 192.168.202.35@tcp1 [ 1532.181794] Key type .llcrypt unregistered [ 1532.184315] Key type ._llcrypt unregistered [ 1551.517545] Key type ._llcrypt registered [ 1551.519793] Key type .llcrypt registered [ 1552.791622] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 19:26:05 (1781220365) [ 1554.242293] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1556.172503] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 19:26:09 (1781220369) [ 1556.977646] Key type .llcrypt unregistered [ 1556.982137] Key type ._llcrypt unregistered [ 1565.502913] Key type ._llcrypt registered [ 1565.509620] Key type .llcrypt registered [ 1565.835395] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1565.859652] alg: No test for adler32 (adler32-zlib) [ 1566.816299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1566.862643] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1576.490164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.135@tcp [ 1577.288539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e network_timeout -r 1 [ 1577.321930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e network_timeout -r 1 [ 1577.355870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.135@tcp [ 1577.373238] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 1577.619574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1577.666927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1577.715469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1577.755921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1577.789423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1577.825696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1577.859888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1577.885127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1577.900340] LNet: 64005:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1577.915274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1577.954988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1577.990822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1578.032331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1578.088318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1578.140257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1578.563712] LNet: 64044:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1578.580986] LNetError: 64044:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1578.608759] LNet: Removed LNI 192.168.202.35@tcp [ 1579.103431] Key type .llcrypt unregistered [ 1579.104917] Key type ._llcrypt unregistered [ 1600.574497] Key type ._llcrypt registered [ 1600.577167] Key type .llcrypt registered [ 1600.761454] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1600.782415] alg: No test for adler32 (adler32-zlib) [ 1601.793492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1601.822420] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1610.836417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.135@tcp [ 1621.051829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1621.087383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1621.126577] LNet: Added LNI 192.168.202.35@tcp1 [8/256/0/180] [ 1622.753248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e network_timeout -r 1 [ 1622.785495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp1 -m GET -e network_timeout -r 1 [ 1622.830250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e network_timeout -r 1 [ 1622.900961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp1 -m GET -e network_timeout -r 1 [ 1622.952689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp -m GET -e network_timeout -r 1 [ 1623.011393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp1 -m GET -e network_timeout -r 1 [ 1623.059284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp -m GET -e network_timeout -r 1 [ 1623.099503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp1 -m GET -e network_timeout -r 1 [ 1623.150813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.135@tcp [ 1623.181629] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 1623.327508] LNet: 1 local NIs in recovery (showing 1): 192.168.202.35@tcp [ 1623.339538] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.135@tcp [ 1623.516474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1623.590509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1623.643610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1623.706526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1623.786652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1623.879065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1623.956507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1624.024700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1624.068642] LNet: 64927:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1624.107462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1624.150397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1624.194404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1624.242544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1624.324644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1624.354660] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 1624.388756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1629.400986] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 1629.409492] LNet: Skipped 1 previous similar message [ 1629.915244] LNet: 65000:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1629.932595] LNetError: 65000:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1629.943657] LNet: Removed LNI 192.168.202.35@tcp [ 1629.962451] LNet: Removed LNI 192.168.202.35@tcp1 [ 1630.480444] Key type .llcrypt unregistered [ 1630.483189] Key type ._llcrypt unregistered [ 1649.691397] Key type ._llcrypt registered [ 1649.693302] Key type .llcrypt registered [ 1651.222871] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 19:27:44 (1781220464) [ 1651.896306] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1651.908564] alg: No test for adler32 (adler32-zlib) [ 1652.841969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1652.971628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1653.008294] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1653.811661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1653.835293] LNet: Added LNI 192.168.202.35@tcp1 [8/256/0/180] [ 1654.932661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 1655.040755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1655.073614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1655.102952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1655.142666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1655.180494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1655.214807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.35@tcp [ 1655.236778] LNetError: 66101:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.202.35@tcp to recovery [ 1655.967353] LNet: 1 local NIs in recovery (showing 1): 192.168.202.35@tcp [ 1655.969935] LNet: There was an unexpected network error while writing to 192.168.202.35: rc = -22 [ 1658.015433] LNet: There was an unexpected network error while writing to 192.168.202.35: rc = -22 [ 1658.909271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1662.111749] LNet: There was an unexpected network error while writing to 192.168.202.35: rc = -22 [ 1662.538096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1662.565275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1662.630292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1662.682214] LNet: 66208:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1662.688884] LNetError: 66208:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1662.698037] LNet: Removed LNI 192.168.202.35@tcp [ 1662.713721] LNet: Removed LNI 192.168.202.35@tcp1 [ 1662.754870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1662.850099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1662.904266] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1663.679219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1663.700813] LNet: Added LNI 192.168.202.35@tcp1 [8/256/0/180] [ 1664.755566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 1664.806560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1664.938934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1665.016666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1665.079527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1665.138432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1665.169560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1665.214961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.35@tcp [ 1665.256211] LNetError: 66481:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.202.35@tcp to recovery [ 1665.895347] LNet: There was an unexpected network error while writing to 192.168.202.35: rc = -22 [ 1668.980779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1671.008232] LNet: There was an unexpected network error while writing to 192.168.202.35: rc = -22 [ 1671.022561] LNet: Skipped 1 previous similar message [ 1676.083334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1676.142321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1676.174434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1684.293947] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 19:28:16 (1781220496) [ 1684.736649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1684.808740] LNet: 67191:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1684.823542] LNetError: 67191:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1684.828193] LNetError: 67191:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1684.834191] LNet: Removed LNI 192.168.202.35@tcp [ 1684.868204] LNet: Removed LNI 192.168.202.35@tcp1 [ 1684.921771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1685.004591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1685.026539] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1685.836441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1685.858258] LNet: Added LNI 192.168.202.35@tcp1 [8/256/0/180] [ 1686.836785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 1686.941485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1687.001365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1687.045797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1687.087585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1687.131588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1687.179790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.35@tcp --health 0 [ 1687.291380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1688.032807] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.35@tcp [ 1688.035797] LNet: There was an unexpected network error while writing to 192.168.202.35: rc = -22 [ 1688.039909] LNet: Skipped 1 previous similar message [ 1703.141786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1703.340348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1703.433155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1703.493495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.35@tcp --health 500 [ 1703.621707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1710.335084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1710.426202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1710.473631] LNet: 67683:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1710.481260] LNetError: 67683:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1710.491330] LNetError: 67683:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1710.498054] LNet: Removed LNI 192.168.202.35@tcp [ 1710.522894] LNet: Removed LNI 192.168.202.35@tcp1 [ 1710.560228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1710.604233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1710.635311] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1711.168587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1711.182742] LNet: Added LNI 192.168.202.35@tcp1 [8/256/0/180] [ 1711.898843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 1711.928586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1711.972847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1711.996490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1712.021167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1712.047980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1712.081928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1712.116686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.35@tcp --health 0 [ 1713.631854] LNet: There was an unexpected network error while writing to 192.168.202.35: rc = -22 [ 1713.652330] LNet: Skipped 3 previous similar messages [ 1714.661867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1725.204361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1725.243853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1725.279298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1733.082925] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 19:29:06 (1781220546) [ 1733.797612] LNet: 68741:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1733.802433] LNetError: 68741:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1733.808371] LNetError: 68741:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1733.817487] LNet: Removed LNI 192.168.202.35@tcp [ 1733.836633] LNet: Removed LNI 192.168.202.35@tcp1 [ 1734.644088] Key type .llcrypt unregistered [ 1734.650365] Key type ._llcrypt unregistered [ 1743.167979] Key type ._llcrypt registered [ 1743.175203] Key type .llcrypt registered [ 1743.410798] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1743.428635] alg: No test for adler32 (adler32-zlib) [ 1744.482986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1744.514689] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1754.597844] Lustre: DEBUG MARKER: Initial discovery [ 1754.749563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.135@tcp [ 1758.377899] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1758.483825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1758.519972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.135@tcp [ 1758.541950] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 1758.568775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1758.611838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.202.135@tcp -r 1 -m GET -l 3 [ 1758.686777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.202.135@tcp -r 1 -m GET -e local_error [ 1758.721252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.202.135@tcp -r 1 -m PUT -l 6 [ 1758.879178] LNet: 1 local NIs in recovery (showing 1): 192.168.202.35@tcp [ 1760.196935] Lustre: DEBUG MARKER: Force oleg235-client.virtnet to discover 192.168.202.135@tcp (in background) [ 1760.313193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.135@tcp [ 1761.793201] Lustre: DEBUG MARKER: Force oleg235-server to discover 192.168.202.35@tcp [ 1763.361436] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 1764.837425] Lustre: DEBUG MARKER: Wait for 69497 [ 1766.521302] Lustre: DEBUG MARKER: Finished wait on 69497 [ 1766.747145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1766.787769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1766.791049] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 1767.094575] LNet: 69957:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1767.101033] LNetError: 69957:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1767.122061] LNet: Removed LNI 192.168.202.35@tcp [ 1767.778579] Key type .llcrypt unregistered [ 1767.787470] Key type ._llcrypt unregistered [ 1785.910229] Key type ._llcrypt registered [ 1785.914294] Key type .llcrypt registered [ 1787.415578] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 19:30:00 (1781220600) [ 1788.015647] Key type .llcrypt unregistered [ 1788.020169] Key type ._llcrypt unregistered [ 1794.566903] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1794.752263] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1794.771045] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1794.778242] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1797.413569] Key type ._llcrypt registered [ 1797.417774] Key type .llcrypt registered [ 1797.688824] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1797.713818] alg: No test for adler32 (adler32-zlib) [ 1798.652202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1798.720266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1798.745514] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1799.814074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1799.829509] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1809.114762] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 19:30:22 (1781220622) [ 1809.741458] LNet: 72035:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1809.750555] LNetError: 72035:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1809.758951] LNet: Removed LNI 192.168.202.35@tcp [ 1809.775273] LNet: Removed LNI 10.1.2.3@tcp [ 1810.616354] Key type .llcrypt unregistered [ 1810.617845] Key type ._llcrypt unregistered [ 1817.647690] Key type ._llcrypt registered [ 1817.649675] Key type .llcrypt registered [ 1817.845367] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1817.859423] alg: No test for adler32 (adler32-zlib) [ 1818.951562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1819.059988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1819.088517] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1819.961258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1819.988065] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1820.937478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.202.35@tcp [ 1821.984688] LNetError: Unexpected error -22 connecting to 192.168.202.35@tcp at host 192.168.202.35:988 [ 1828.213720] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 19:30:41 (1781220641) [ 1828.679769] LNet: 73168:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1828.693805] LNetError: 73168:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1828.701439] LNet: Removed LNI 192.168.202.35@tcp [ 1828.720569] LNet: Removed LNI 10.1.2.3@tcp [ 1829.233620] Key type .llcrypt unregistered [ 1829.238575] Key type ._llcrypt unregistered [ 1835.268678] Key type ._llcrypt registered [ 1835.270620] Key type .llcrypt registered [ 1835.551528] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1835.572873] alg: No test for adler32 (adler32-zlib) [ 1836.571455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1836.667476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1836.731466] LNet: Added LNI 192.168.202.35@tcp1 [8/256/0/180] [ 1837.658824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1837.670158] LNet: Added LNI 192.168.202.35@tcp2 [8/256/0/180] [ 1838.676459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.202.35@tcp1 --nid 192.168.202.35@tcp2 [ 1843.436466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.35@tcp1 192.168.202.35@tcp2 [ 1843.487869] LNetError: 74003:0:(lib-move.c:2007:lnet_handle_find_routed_path()) Unable to send message from 192.168.202.35@tcp1 to 192.168.202.35@tcp2 - Route table may be misconfigured [ 1843.547791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.35@tcp2 192.168.202.35@tcp1 [ 1851.190683] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 19:31:03 (1781220663) [ 1851.351736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1851.384168] LNet: 74571:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1851.393906] LNetError: 74571:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1851.404701] LNet: Removed LNI 192.168.202.35@tcp1 [ 1852.459799] LNet: Removed LNI 192.168.202.35@tcp2 [ 1852.512660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1852.661142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1852.719917] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1853.577270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1853.594775] LNet: Added LNI 192.168.202.35@tcp1 [8/256/0/180] [ 1854.717786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 1854.769146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.35@tcp 192.168.202.35@tcp [ 1854.816768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.35@tcp1 192.168.202.35@tcp1 [ 1854.918301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.35@tcp -d 192.168.202.35@tcp -e network_timeout [ 1854.995682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.35@tcp -d 192.168.202.35@tcp1 -e network_timeout [ 1855.098123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp -e network_timeout [ 1855.156650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp1 -e network_timeout [ 1855.234912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 1855.263752] LNet: There was an unexpected network error while writing to 192.168.202.35: rc = -22 [ 1855.283497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1855.420264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1855.555793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1855.653169] LNet: 1 local NIs in recovery (showing 1): 192.168.202.35@tcp [ 1863.041154] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 19:31:16 (1781220676) [ 1863.362801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1863.427114] LNet: 75438:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1863.436568] LNetError: 75438:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1863.446362] LNetError: 75438:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1863.452898] LNet: Removed LNI 192.168.202.35@tcp [ 1864.486346] LNet: Removed LNI 192.168.202.35@tcp1 [ 1864.529803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1864.654960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1865.080847] LNet: 75475:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1865.526629] Key type .llcrypt unregistered [ 1865.531621] Key type ._llcrypt unregistered [ 1877.539431] Key type ._llcrypt registered [ 1877.541245] Key type .llcrypt registered [ 1878.825196] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 19:31:31 (1781220691) [ 1879.419821] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1879.435813] alg: No test for adler32 (adler32-zlib) [ 1880.405845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1881.866882] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1883.971086] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 19:31:36 (1781220696) [ 1884.305504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1884.348811] LNet: 76573:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1884.393222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1884.474563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1884.509403] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1885.101394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1885.120745] LNet: Added LNI 192.168.202.35@tcp1 [8/256/0/180] [ 1886.002168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 1886.051163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp1 [ 1886.099100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp1 [ 1893.550380] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 19:31:46 (1781220706) [ 1895.840486] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg235-server [ 1897.910553] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 19:31:50 (1781220710) [ 1900.482827] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg235-server [ 1903.240457] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 19:31:55 (1781220715) [ 1905.510532] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg235-server [ 1907.511491] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 19:32:00 (1781220720) [ 1909.262830] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg235-server [ 1911.565714] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 19:32:04 (1781220724) [ 1913.816879] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg235-server [ 1916.185950] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 19:32:08 (1781220728) [ 1918.431329] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg235-server [ 1920.620595] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 19:32:13 (1781220733) [ 1923.434800] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg235-server [ 1925.547691] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 19:32:18 (1781220738) [ 1927.463808] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg235-server [ 1929.584496] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 19:32:22 (1781220742) [ 1932.144778] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg235-server [ 1933.839374] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 19:32:26 (1781220746) [ 1934.081562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1934.099199] LNet: 80451:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1934.107796] LNetError: 80451:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1935.135326] LNet: Removed LNI 192.168.202.35@tcp [ 1935.140539] LNetError: 80451:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1935.166775] LNet: Removed LNI 192.168.202.35@tcp1 [ 1935.224542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1935.289991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1935.338552] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1936.192428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1936.348933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 1936.668541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1936.701392] LNet: 80599:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1936.724658] LNetError: 80599:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1936.759462] LNet: Removed LNI 192.168.202.35@tcp [ 1936.817228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1936.946614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1936.998426] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1938.044259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 1938.312684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1939.369599] LNet: Removed LNI 192.168.202.35@tcp [ 1939.419202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1939.506507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1939.540619] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1940.395684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1940.478628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 1940.742287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1940.774732] LNet: 80891:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1940.792921] LNet: 80891:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1940.803846] LNetError: 80891:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1940.817628] LNetError: 80891:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1940.877293] LNet: Removed LNI 192.168.202.35@tcp [ 1940.901858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1941.006599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1941.059178] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1942.205416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 1942.528924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1943.596828] LNet: Removed LNI 192.168.202.35@tcp [ 1943.629805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1943.718821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1943.773700] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1944.687523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1944.814535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 1944.962263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1944.979447] LNet: 81184:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1944.983583] LNet: 81184:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1944.988859] LNetError: 81184:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1944.993582] LNetError: 81184:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1945.014283] LNet: Removed LNI 192.168.202.35@tcp [ 1945.039751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1945.088856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1945.127983] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1945.783749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 1946.056818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1947.109700] LNet: Removed LNI 192.168.202.35@tcp [ 1947.148763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1947.221775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1947.275866] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1948.224210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1948.366993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 1948.570372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1949.606856] LNet: Removed LNI 192.168.202.35@tcp [ 1949.667020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1949.751730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1949.770537] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1950.480176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 1950.783695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1951.847437] LNet: Removed LNI 192.168.202.35@tcp [ 1951.894503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1951.992548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1952.032399] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1952.972293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1953.111470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 1953.303672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1953.326584] LNet: 81769:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1953.331427] LNet: 81769:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1953.338857] LNetError: 81769:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1953.347943] LNetError: 81769:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1953.362642] LNet: Removed LNI 192.168.202.35@tcp [ 1953.379794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1953.435466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1953.469424] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1954.177315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 1955.629212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1956.712624] LNet: Removed LNI 192.168.202.35@tcp [ 1956.768288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1956.850471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1956.890561] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1957.719294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1957.800547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 1958.064566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1959.151871] LNet: Removed LNI 192.168.202.35@tcp [ 1959.209910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1959.319407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1959.363326] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1960.611766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 1962.191588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1963.302825] LNet: Removed LNI 192.168.202.35@tcp [ 1963.356297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1963.447430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1963.466665] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1964.603850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1964.717792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1964.769805] LNet: Removed LNI 192.168.202.35@tcp [ 1964.802730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1964.877456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1964.898361] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1965.918763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1974.620035] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 19:33:07 (1781220787) [ 1974.825267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1974.871857] LNet: 83063:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1974.877656] LNet: 83063:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1974.887251] LNetError: 83063:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1974.895422] LNetError: 83063:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1974.926640] LNet: Removed LNI 192.168.202.35@tcp [ 1974.947434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1975.006971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1975.039381] LNet: Added LNI 192.168.202.35@tcp231 [8/256/0/180] [ 1976.715875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1976.761049] LNet: Removed LNI 192.168.202.35@tcp231 [ 1976.793957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1976.879586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1976.916343] LNet: Added LNI 192.168.202.35@tcp231 [8/256/0/180] [ 1976.936377] LNet: 83210:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1977.548220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1977.591297] LNet: Removed LNI 192.168.202.35@tcp231 [ 1977.618376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1977.655718] LNet: Added LNI 192.168.202.35@tcp231 [8/256/0/180] [ 1979.497707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1979.585971] LNet: Removed LNI 192.168.202.35@tcp231 [ 1979.612255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1979.731949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1979.776649] LNet: Added LNI 192.168.202.35@tcp231 [8/256/0/0] [ 1979.789687] LNet: 83362:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1980.443221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1980.493240] LNet: Removed LNI 192.168.202.35@tcp231 [ 1980.518670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1980.559576] LNet: Added LNI 192.168.202.35@tcp231 [8/256/0/0] [ 1981.686158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1981.737426] LNet: Removed LNI 192.168.202.35@tcp231 [ 1981.751582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1981.847457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1981.874635] LNet: Added LNI 192.168.202.35@tcp231 [8/256/0/60] [ 1981.878592] LNet: 83514:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1982.334575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1982.376488] LNet: Removed LNI 192.168.202.35@tcp231 [ 1982.394471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1982.435319] LNet: Added LNI 192.168.202.35@tcp231 [8/256/0/60] [ 1983.545216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1983.592684] LNet: Removed LNI 192.168.202.35@tcp231 [ 1983.618063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1983.675385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1983.696590] LNet: Added LNI 192.168.202.35@tcp231 [8/256/0/180] [ 1984.052400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1984.091861] LNet: Removed LNI 192.168.202.35@tcp231 [ 1984.117238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1984.144590] LNet: Added LNI 192.168.202.35@tcp231 [8/256/0/180] [ 1994.140539] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 19:33:27 (1781220807) [ 1994.283109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1994.317365] LNet: Removed LNI 192.168.202.35@tcp231 [ 1994.354522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1994.459872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1994.491942] LNet: Added LNI 192.168.202.35@tcp232 [8/256/0/180] [ 1995.363189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1995.420852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1995.456354] LNet: Removed LNI 192.168.202.35@tcp232 [ 1995.492918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1995.551401] LNet: Added LNI 192.168.202.35@tcp232 [8/256/0/180] [ 1996.753629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1996.837669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1996.882161] LNet: Removed LNI 192.168.202.35@tcp232 [ 1996.916961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1996.964880] LNet: Added LNI 192.168.202.35@tcp232 [8/256/0/180] [ 1997.654374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1997.696394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1997.734073] LNet: Removed LNI 192.168.202.35@tcp232 [ 1997.752278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1997.780336] LNet: Added LNI 192.168.202.35@tcp232 [8/256/0/180] [ 1998.770245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1998.837393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1998.876117] LNet: Removed LNI 192.168.202.35@tcp232 [ 1998.922982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 2005.938326] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 19:33:38 (1781220818) [ 2007.440809] Key type .llcrypt unregistered [ 2007.446201] Key type ._llcrypt unregistered [ 2014.125084] Key type ._llcrypt registered [ 2014.129156] Key type .llcrypt registered [ 2014.404156] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2014.436845] alg: No test for adler32 (adler32-zlib) [ 2015.424993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2015.454279] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2022.850635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.135@tcp [ 2032.007913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2032.063979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2032.124032] LNet: Added LNI 192.168.202.35@tcp1 [8/256/0/180] [ 2033.841433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e local_interrupt -r 2 [ 2033.910450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp1 -m GET -e local_interrupt -r 2 [ 2033.971104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e local_interrupt -r 2 [ 2034.023869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp1 -m GET -e local_interrupt -r 2 [ 2034.067735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp -m GET -e local_interrupt -r 2 [ 2034.114556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp1 -m GET -e local_interrupt -r 2 [ 2034.154634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp -m GET -e local_interrupt -r 2 [ 2034.198517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp1 -m GET -e local_interrupt -r 2 [ 2034.252839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.35@tcp 192.168.202.135@tcp [ 2034.286740] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 2034.292120] LNet: 1 local NIs in recovery (showing 1): 192.168.202.35@tcp [ 2034.550900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2034.579105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2034.608659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2034.639555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2034.670871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2034.705787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2034.737749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2034.788333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2034.812851] LNet: 86175:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2034.835746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2034.884609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2034.965344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2035.034608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2035.068650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2035.100032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2035.676358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e local_dropped -r 2 [ 2035.738452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp1 -m GET -e local_dropped -r 2 [ 2035.801799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e local_dropped -r 2 [ 2035.866284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp1 -m GET -e local_dropped -r 2 [ 2035.947588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp -m GET -e local_dropped -r 2 [ 2036.020614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp1 -m GET -e local_dropped -r 2 [ 2036.087853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp -m GET -e local_dropped -r 2 [ 2036.138959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp1 -m GET -e local_dropped -r 2 [ 2036.180838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.35@tcp 192.168.202.135@tcp [ 2036.201690] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 2036.538428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2036.591499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2036.629776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2036.662522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2036.691080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2036.732393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2036.770935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2036.805180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2036.834807] LNet: 86296:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2036.871575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2036.919977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2036.954876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2036.995669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2037.029835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2037.066920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2037.667740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e local_aborted -r 2 [ 2037.725039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp1 -m GET -e local_aborted -r 2 [ 2037.817691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e local_aborted -r 2 [ 2037.889320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp1 -m GET -e local_aborted -r 2 [ 2037.951915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp -m GET -e local_aborted -r 2 [ 2038.002351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp1 -m GET -e local_aborted -r 2 [ 2038.039470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp -m GET -e local_aborted -r 2 [ 2038.073742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp1 -m GET -e local_aborted -r 2 [ 2038.106467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.35@tcp 192.168.202.135@tcp [ 2038.134808] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 2038.388801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2038.417848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2038.442918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2038.467067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2038.511566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2038.558241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2038.594943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2038.628314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2038.657557] LNet: 86416:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2038.673522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2038.710202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2038.774113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2038.816835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2038.861867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2038.902327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2039.354767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e local_no_route -r 2 [ 2039.383350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp1 -m GET -e local_no_route -r 2 [ 2039.415853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e local_no_route -r 2 [ 2039.447727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp1 -m GET -e local_no_route -r 2 [ 2039.475619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp -m GET -e local_no_route -r 2 [ 2039.520798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp1 -m GET -e local_no_route -r 2 [ 2039.573960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp -m GET -e local_no_route -r 2 [ 2039.622924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp1 -m GET -e local_no_route -r 2 [ 2039.677897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.35@tcp 192.168.202.135@tcp [ 2039.915397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2039.977299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2040.014321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2040.068772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2040.094502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2040.118706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2040.139395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2040.157468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2040.188987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2040.228428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2040.287952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2040.352821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2040.389190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2040.462134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2041.080765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e local_timeout -r 2 [ 2041.109083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp1 -m GET -e local_timeout -r 2 [ 2041.142477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e local_timeout -r 2 [ 2041.168714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp1 -m GET -e local_timeout -r 2 [ 2041.193809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp -m GET -e local_timeout -r 2 [ 2041.218260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp1 -m GET -e local_timeout -r 2 [ 2041.246920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp -m GET -e local_timeout -r 2 [ 2041.273624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp1 -m GET -e local_timeout -r 2 [ 2041.302944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.35@tcp 192.168.202.135@tcp [ 2041.318508] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 2041.324353] LNet: Skipped 1 previous similar message [ 2041.581516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2041.623789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2041.655679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2041.688442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2041.721522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2041.757467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2041.785870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2041.822483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2041.846080] LNet: 86656:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2041.849739] LNet: 86656:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 2041.876187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2041.928336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2042.007717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2042.062983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2042.131869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2042.176947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2042.695657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e remote_dropped -r 2 [ 2042.723836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp1 -m GET -e remote_dropped -r 2 [ 2042.761769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e remote_dropped -r 2 [ 2042.795537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp1 -m GET -e remote_dropped -r 2 [ 2042.829469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp -m GET -e remote_dropped -r 2 [ 2042.868546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp1 -m GET -e remote_dropped -r 2 [ 2042.892767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp -m GET -e remote_dropped -r 2 [ 2042.920843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp1 -m GET -e remote_dropped -r 2 [ 2042.979600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.35@tcp 192.168.202.135@tcp [ 2043.005509] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.135@tcp [ 2043.234579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2043.257481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2043.285773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2043.315780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2043.350331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2043.376198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2043.407328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2043.444432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2043.479962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2043.537589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2043.594579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2043.629486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2043.665405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2043.710474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2049.225789] LNet: 86850:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2049.232405] LNetError: 86850:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2050.271196] LNet: Removed LNI 192.168.202.35@tcp [ 2050.276627] LNetError: 86850:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2050.289938] LNet: Removed LNI 192.168.202.35@tcp1 [ 2050.846248] Key type .llcrypt unregistered [ 2050.848869] Key type ._llcrypt unregistered [ 2069.241415] Key type ._llcrypt registered [ 2069.243372] Key type .llcrypt registered [ 2070.595809] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 19:34:43 (1781220883) [ 2071.180242] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2071.202363] alg: No test for adler32 (adler32-zlib) [ 2072.109836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2072.150349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2072.178863] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2073.151538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 2073.182925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.35@tcp [ 2073.201631] LNetError: 87829:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.202.35@tcp to recovery [ 2073.286638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.202.35@tcp [ 2073.403823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2080.726330] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 19:34:53 (1781220893) [ 2080.837626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2080.870370] LNet: 88414:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2080.884903] LNetError: 88414:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2080.923073] LNet: Removed LNI 192.168.202.35@tcp [ 2080.959205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2081.036429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2081.071072] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2081.916450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2081.957277] LNet: Added LNI 192.168.202.35@tcp1 [8/256/0/180] [ 2083.341357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.202.35@tcp [ 2083.383990] LNetError: 88687:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.202.35@tcp to recovery [ 2084.063548] LNet: 1 local NIs in recovery (showing 1): 192.168.202.35@tcp [ 2084.072583] LNet: There was an unexpected network error while writing to 192.168.202.35: rc = -22 [ 2084.786095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.202.35@tcp [ 2094.464493] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 19:35:07 (1781220907) [ 2096.747604] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg235-server [ 2098.488695] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 19:35:11 (1781220911) [ 2101.254897] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg235-server [ 2103.452035] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 19:35:16 (1781220916) [ 2105.613157] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg235-server [ 2107.570535] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 19:35:20 (1781220920) [ 2107.892942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2107.921399] LNet: 90318:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2107.935209] LNetError: 90318:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2107.940403] LNet: Removed LNI 192.168.202.35@tcp [ 2107.958724] LNet: Removed LNI 192.168.202.35@tcp1 [ 2107.986937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2108.093934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2108.119175] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2109.200614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 2109.285354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 2109.329114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2109.414850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 2109.447573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2109.490244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 2116.688845] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 19:35:29 (1781220929) [ 2117.237993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2117.265196] LNet: 91044:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2117.272736] LNetError: 91044:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2117.278682] LNetError: 91044:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2117.301087] LNet: Removed LNI 192.168.202.35@tcp [ 2117.319672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2117.390842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2117.412730] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2124.521818] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 19:35:37 (1781220937) [ 2124.792298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2124.821450] LNet: 91747:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2124.831329] LNetError: 91747:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2124.859830] LNet: Removed LNI 192.168.202.35@tcp [ 2124.889971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2124.975847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2125.011705] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2133.678707] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 19:35:46 (1781220946) [ 2135.923695] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 2137.864367] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 19:35:50 (1781220950) [ 2138.670915] LNet: 92782:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2138.681460] LNetError: 92782:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2138.702469] LNet: Removed LNI 192.168.202.35@tcp [ 2139.910788] Key type .llcrypt unregistered [ 2139.916585] Key type ._llcrypt unregistered [ 2148.148563] Key type ._llcrypt registered [ 2148.150934] Key type .llcrypt registered [ 2148.420290] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2148.449785] alg: No test for adler32 (adler32-zlib) [ 2149.430722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2149.463452] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2158.105591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.135@tcp [ 2163.603676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.202.135@tcp [ 2164.141528] LNet: 93199:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2164.157446] LNetError: 93199:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2164.170885] LNet: Removed LNI 192.168.202.35@tcp [ 2164.727783] Key type .llcrypt unregistered [ 2164.729899] Key type ._llcrypt unregistered [ 2179.274561] Key type ._llcrypt registered [ 2179.278150] Key type .llcrypt registered [ 2180.770693] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 2182.456486] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 2184.771174] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 19:36:37 (1781220997) [ 2187.390594] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 2189.653278] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 19:36:42 (1781221002) [ 2191.953911] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg235-server [ 2194.673211] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 19:36:47 (1781221007) [ 2196.938268] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg235-server [ 2199.552330] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 19:36:52 (1781221012) [ 2202.113734] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2202.178770] alg: No test for adler32 (adler32-zlib) [ 2203.455117] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 2203.868681] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2205.671184] Key type lgssc registered [ 2207.516436] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2231.461170] Key type lgssc unregistered [ 2231.802479] LNet: 96385:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2231.814507] LNetError: 96385:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2231.835800] LNet: Removed LNI 192.168.202.35@tcp [ 2232.742779] Key type .llcrypt unregistered [ 2232.747075] Key type ._llcrypt unregistered [ 2249.203155] Key type ._llcrypt registered [ 2249.204996] Key type .llcrypt registered [ 2250.564331] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 19:37:43 (1781221063) [ 2251.278889] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2251.306821] alg: No test for adler32 (adler32-zlib) [ 2252.264356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2252.349575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2252.409552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2252.446527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2252.492660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2259.972093] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 19:37:52 (1781221072) [ 2260.272911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2260.315961] LNet: 97800:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2260.368075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2260.469651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2260.494955] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2261.679489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2261.727205] LNet: 97944:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2261.733821] LNetError: 97944:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2261.774234] LNet: Removed LNI 192.168.202.35@tcp [ 2261.806256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2261.889941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2261.907705] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2262.389153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2262.417181] LNetError: 97972:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2262.429414] LNet: Removed LNI 192.168.202.35@tcp [ 2262.451895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2262.546959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2262.604165] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2271.946688] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 19:38:04 (1781221084) [ 2272.107582] LNet: 98555:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2272.121245] LNet: 98555:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2272.146180] LNetError: 98555:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2272.215313] LNet: Removed LNI 192.168.202.35@tcp [ 2273.343775] Key type .llcrypt unregistered [ 2273.347486] Key type ._llcrypt unregistered [ 2283.285427] Key type ._llcrypt registered [ 2283.287072] Key type .llcrypt registered [ 2283.654592] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2283.688095] alg: No test for adler32 (adler32-zlib) [ 2284.842546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2285.075743] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2285.077781] LNet: *** cfs_fail_loc=e003, val=100*** [ 2286.860323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 2286.934574] LNet: *** cfs_fail_loc=e003, val=100*** [ 2286.941378] LNet: Skipped 2 previous similar messages [ 2286.986416] Ping reply received with mapping: [ 2286.988306] found metadata reply NI[192.168.202.35@tcp] [ 2287.001064] 23 ca a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2287.005636] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2287.010385] 5a 5a 5a 5a 5a 5a 5a 5a [ 2287.310646] LNet: 98962:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2287.325485] LNetError: 98962:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2287.360748] LNet: Removed LNI 192.168.202.35@tcp [ 2288.586502] Key type .llcrypt unregistered [ 2288.589162] Key type ._llcrypt unregistered [ 2304.379745] Key type ._llcrypt registered [ 2304.381468] Key type .llcrypt registered [ 2306.052960] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 19:38:38 (1781221118) [ 2307.095372] Key type .llcrypt unregistered [ 2307.098885] Key type ._llcrypt unregistered [ 2314.830462] Key type ._llcrypt registered [ 2314.831855] Key type .llcrypt registered [ 2315.022585] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2315.041868] alg: No test for adler32 (adler32-zlib) [ 2316.153562] LNetError: 100009:0:(api-ni.c:2729:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2316.608571] Key type .llcrypt unregistered [ 2316.610763] Key type ._llcrypt unregistered [ 2323.327840] Key type ._llcrypt registered [ 2323.335347] Key type .llcrypt registered [ 2325.441674] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 19:38:57 (1781221137) [ 2326.058372] Key type .llcrypt unregistered [ 2326.065630] Key type ._llcrypt unregistered [ 2333.963540] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2333.968189] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2336.800233] Key type ._llcrypt registered [ 2336.803774] Key type .llcrypt registered [ 2337.240859] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2337.289625] alg: No test for adler32 (adler32-zlib) [ 2338.282547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2338.403284] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2350.044285] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 19:39:22 (1781221162) [ 2350.145723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2350.171516] LNet: 101782:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2350.187300] LNetError: 101782:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2350.224521] LNet: Removed LNI 10.1.2.3@tcp [ 2350.260446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2350.960566] Key type .llcrypt unregistered [ 2350.964255] Key type ._llcrypt unregistered [ 2360.184031] Key type ._llcrypt registered [ 2360.187087] Key type .llcrypt registered [ 2360.455229] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2360.474947] alg: No test for adler32 (adler32-zlib) [ 2361.569726] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2361.667476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 2361.912799] LNet: 102078:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2361.941726] LNetError: 102078:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2361.974901] LNet: Removed LNI 192.168.202.35@tcp [ 2362.880612] Key type .llcrypt unregistered [ 2362.891202] Key type ._llcrypt unregistered [ 2376.859564] Key type ._llcrypt registered [ 2376.863327] Key type .llcrypt registered [ 2378.317675] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 19:39:51 (1781221191) [ 2379.248361] Key type .llcrypt unregistered [ 2379.250885] Key type ._llcrypt unregistered [ 2388.196681] Key type ._llcrypt registered [ 2388.201614] Key type .llcrypt registered [ 2388.511775] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2388.549570] alg: No test for adler32 (adler32-zlib) [ 2397.838910] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 19:40:10 (1781221210) [ 2398.205352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2398.258292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2398.367301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2398.402440] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2399.409771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2399.454875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2399.514832] LNet: Removed LNI 192.168.202.35@tcp [ 2399.998383] LNet: 103824:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2400.420390] Key type .llcrypt unregistered [ 2400.422789] Key type ._llcrypt unregistered [ 2413.590421] Key type ._llcrypt registered [ 2413.593505] Key type .llcrypt registered [ 2415.032605] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 19:40:27 (1781221227) [ 2415.602638] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2415.622709] alg: No test for adler32 (adler32-zlib) [ 2416.595825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2416.675976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2416.726746] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2417.732844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 2417.814546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.202.35@tcp [ 2417.840863] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2417.845775] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2417.848576] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2417, 0, 0 [ 2417.858642] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2417.862112] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2417.869778] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2417.873921] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2417, 0, 0 [ 2417.878307] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2417.880822] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2417.888545] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2417.896125] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2417, 0, 2467 [ 2417.905180] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2417.906945] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2417.913311] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2417.916447] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2417, 0, 2467 [ 2417.922312] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2425.178188] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 19:40:38 (1781221238) [ 2425.956945] LNet: 105364:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2425.970744] LNetError: 105364:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2425.993154] LNet: Removed LNI 192.168.202.35@tcp [ 2426.795847] Key type .llcrypt unregistered [ 2426.802258] Key type ._llcrypt unregistered [ 2434.142890] Key type ._llcrypt registered [ 2434.157893] Key type .llcrypt registered [ 2434.502933] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2434.533664] alg: No test for adler32 (adler32-zlib) [ 2435.519220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2435.583969] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2444.023202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.135@tcp [ 2454.730116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2454.759965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2454.774845] LNet: Added LNI 192.168.202.35@tcp1 [8/256/0/180] [ 2455.589379] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2455.703553] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2455.709067] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2455.720697] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2458.180543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2458.229662] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2471.995604] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 19:41:25 (1781221285) [ 2472.631048] LNet: 106871:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2472.641687] LNetError: 106871:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2472.666651] LNet: Removed LNI 192.168.202.35@tcp [ 2473.695221] LNet: Removed LNI 192.168.202.35@tcp1 [ 2473.706345] LNetError: 106871:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2473.718309] LNetError: 106871:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2473.734642] LNet: Removed LNI 10.1.2.3@tcp99 [ 2474.373223] Key type .llcrypt unregistered [ 2474.376375] Key type ._llcrypt unregistered [ 2482.501356] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2482.507090] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2485.108900] Key type ._llcrypt registered [ 2485.111162] Key type .llcrypt registered [ 2485.337799] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2485.377193] alg: No test for adler32 (adler32-zlib) [ 2486.350781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2486.439195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2486.472525] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2487.028644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2487.049552] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2488.818960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2488.845761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2488.872947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2488.886290] LNetError: 107615:0:(peer.c:2175:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2488.915443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2488.942153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 2489.390775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2489.492464] LNet: 107636:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2489.497467] LNetError: 107636:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2489.508369] LNet: Removed LNI 192.168.202.35@tcp [ 2489.525308] LNet: Removed LNI 10.1.2.3@tcp [ 2490.285493] Key type .llcrypt unregistered [ 2490.287931] Key type ._llcrypt unregistered [ 2501.440741] Key type ._llcrypt registered [ 2501.441968] Key type .llcrypt registered [ 2502.740280] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 19:41:55 (1781221315) [ 2503.165538] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2503.176612] alg: No test for adler32 (adler32-zlib) [ 2504.088689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2504.152270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2504.175183] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2504.999905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 2505.098874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg235-client@tcp [ 2512.445629] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 19:42:05 (1781221325) [ 2512.680945] LNet: 109170:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2512.690211] LNetError: 109170:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2512.703947] LNet: Removed LNI 192.168.202.35@tcp [ 2513.415800] Key type .llcrypt unregistered [ 2513.420117] Key type ._llcrypt unregistered [ 2521.144162] Key type ._llcrypt registered [ 2521.149252] Key type .llcrypt registered [ 2521.292045] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2521.307362] alg: No test for adler32 (adler32-zlib) [ 2522.272603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2522.306580] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2522.365440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2529.891024] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 19:42:22 (1781221342) [ 2531.494715] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2534.205328] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 19:42:26 (1781221346) [ 2536.163770] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2538.064812] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 19:42:31 (1781221351) [ 2538.165794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2538.195645] LNet: 110584:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2538.201606] LNetError: 110584:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2538.235257] LNet: Removed LNI 192.168.202.35@tcp [ 2538.269611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2538.349680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2538.377579] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2539.224741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2539.328705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2548.609957] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 19:42:40 (1781221360) [ 2548.769985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2548.823848] LNet: 111294:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2548.844687] LNetError: 111294:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2548.869426] LNet: Removed LNI 192.168.202.35@tcp [ 2548.925454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2549.052913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2549.113618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2550.414453] Key type .llcrypt unregistered [ 2550.420300] Key type ._llcrypt unregistered [ 2566.276933] Key type ._llcrypt registered [ 2566.280137] Key type .llcrypt registered [ 2568.952361] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 19:43:00 (1781221380) [ 2569.646810] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2569.680541] alg: No test for adler32 (adler32-zlib) [ 2570.834442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2570.992348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2571.084581] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2572.370425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2572.503383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 2581.052627] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 19:43:13 (1781221393) [ 2581.245681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2581.313039] LNet: 112851:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2581.335879] LNetError: 112851:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2581.360030] LNet: Removed LNI 192.168.202.35@tcp [ 2581.408250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2581.562585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2581.643392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2590.583624] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 19:43:23 (1781221403) [ 2591.069452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2591.097596] LNet: 113437:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2591.130329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2591.214543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2591.736345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2591.776539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2591.891605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2592.451161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2592.488327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2592.592564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2592.620926] LNet: 113470:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2592.640385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2592.699951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2593.119727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2593.141954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2593.205273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2593.561586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2593.610985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2600.911626] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 19:43:33 (1781221413) [ 2601.287272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2601.324539] LNet: 114071:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2601.376853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2601.455064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2601.492948] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2602.486686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2602.515926] LNet: Added LNI 192.168.202.35@tcp2 [8/256/0/180] [ 2603.804409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2603.878413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 2604.192797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2604.216614] LNetError: 114350:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2604.239412] LNet: Removed LNI 192.168.202.35@tcp [ 2604.273237] LNet: Removed LNI 192.168.202.35@tcp2 [ 2604.318795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2604.439552] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2604.447726] LNet: Added LNI 192.168.202.35@tcp2 [8/256/0/180] [ 2604.458431] LNet: 114356:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2605.441538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 2605.682648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2605.696569] LNet: 114393:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2605.700480] LNet: 114393:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2605.705797] LNetError: 114393:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2605.709596] LNetError: 114393:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2605.717319] LNet: Removed LNI 192.168.202.35@tcp [ 2605.734500] LNet: Removed LNI 192.168.202.35@tcp2 [ 2605.764120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2605.814365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2605.836107] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2606.721080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2606.757723] LNet: Added LNI 192.168.202.35@tcp2 [8/256/0/180] [ 2607.759545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 2607.825665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2607.895591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 2608.198966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2608.228973] LNetError: 114678:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2608.236932] LNetError: 114678:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2608.244179] LNet: Removed LNI 192.168.202.35@tcp [ 2608.276795] LNet: Removed LNI 192.168.202.35@tcp2 [ 2608.327527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2608.416739] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2608.429741] LNet: Added LNI 192.168.202.35@tcp2 [8/256/0/180] [ 2608.436361] LNet: 114684:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2609.213614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 2609.596791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2609.628556] LNet: Removed LNI 192.168.202.35@tcp [ 2610.662625] LNet: Removed LNI 192.168.202.35@tcp2 [ 2610.724568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2610.834519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2610.874590] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2611.893889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2611.918917] LNet: Added LNI 192.168.202.35@tcp2 [8/256/0/180] [ 2613.020711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2613.078228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 2613.478551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2613.498421] LNetError: 115000:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2613.503916] LNetError: 115000:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2613.508465] LNet: Removed LNI 192.168.202.35@tcp [ 2613.517256] LNet: Removed LNI 192.168.202.35@tcp2 [ 2613.535914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2613.587067] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2613.593206] LNet: Added LNI 192.168.202.35@tcp2 [8/256/0/180] [ 2613.597310] LNet: 115006:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2614.191917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 2614.495076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2614.526546] LNet: 115044:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2614.535107] LNet: 115044:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2615.583306] LNet: Removed LNI 192.168.202.35@tcp [ 2615.600039] LNet: Removed LNI 192.168.202.35@tcp2 [ 2615.629143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2615.718557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2615.753973] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2616.604371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2616.658695] LNet: Added LNI 192.168.202.35@tcp2 [8/256/0/180] [ 2617.737789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 2617.805524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2617.894750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 2618.132985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2619.167207] LNet: Removed LNI 192.168.202.35@tcp [ 2620.198350] LNet: Removed LNI 192.168.202.35@tcp2 [ 2620.288036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2620.388636] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2620.394440] LNet: Added LNI 192.168.202.35@tcp2 [8/256/0/180] [ 2620.401961] LNet: 115334:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2621.104451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 2621.455661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2621.513280] LNetError: 115372:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2621.518782] LNetError: 115372:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2621.525508] LNet: Removed LNI 192.168.202.35@tcp [ 2622.584651] LNet: Removed LNI 192.168.202.35@tcp2 [ 2622.644848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2622.745879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2622.783710] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2623.667961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2623.690075] LNet: Added LNI 192.168.202.35@tcp2 [8/256/0/180] [ 2624.836098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2624.874900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2624.969371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 2625.531244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2626.591334] LNet: Removed LNI 192.168.202.35@tcp [ 2626.603034] LNet: Removed LNI 192.168.202.35@tcp2 [ 2626.668959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2626.820390] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2626.834471] LNet: Added LNI 192.168.202.35@tcp2 [8/256/0/180] [ 2626.850632] LNet: 115674:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2627.540201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 2628.060662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2629.151266] LNet: Removed LNI 192.168.202.35@tcp [ 2629.169496] LNet: Removed LNI 192.168.202.35@tcp2 [ 2629.251239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2629.392986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2629.479582] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2630.476995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2630.493926] LNet: Added LNI 192.168.202.35@tcp2 [8/256/0/180] [ 2631.546465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 2631.591932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2631.676659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2631.758585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 2632.310959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2632.345162] LNet: 116028:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2632.362435] LNet: 116028:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2633.439579] LNet: Removed LNI 192.168.202.35@tcp [ 2634.475373] LNet: Removed LNI 192.168.202.35@tcp2 [ 2634.542455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2634.730632] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2634.752927] LNet: Added LNI 192.168.202.35@tcp2 [8/256/0/180] [ 2635.751732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 2646.632997] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2649.158256] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 19:44:21 (1781221461) [ 2649.334820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2649.371431] LNetError: 116792:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2649.381178] LNetError: 116792:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2649.391069] LNet: Removed LNI 192.168.202.35@tcp [ 2649.424192] LNet: Removed LNI 192.168.202.35@tcp2 [ 2649.494953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2649.822254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2649.846796] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2650.542166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2650.566067] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2651.803887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2651.849454] LNet: Removed LNI 192.168.202.35@tcp [ 2651.877706] LNet: Removed LNI 10.1.2.3@tcp [ 2651.900835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2651.992487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2652.038904] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2652.049525] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2652.567625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2652.584640] LNet: Removed LNI 192.168.202.35@tcp [ 2652.599203] LNet: Removed LNI 10.1.2.3@tcp [ 2660.715321] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 19:44:33 (1781221473) [ 2661.095147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2661.157355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2661.351413] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2661.500721] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2661.502941] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2661.509868] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2663.941968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2663.978668] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2664.740232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2664.754455] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2666.604772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 2666.640242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2666.703952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.202.35@tcp -m PUT -e local_timeout -r 1 [ 2666.810559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2666.911614] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2666.919423] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2667.158765] LNetError: 117736:0:(lib-move.c:4710:lnet_parse()) 192.168.202.35@tcp, src 192.168.202.35@tcp: Dropping PUT (error -108 looking up sender) [ 2668.194666] LNet: 118194:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2668.218337] LNet: 118194:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2669.279354] LNet: Removed LNI 192.168.202.35@tcp [ 2669.299824] LNet: Removed LNI 10.1.2.3@tcp [ 2669.747244] Key type .llcrypt unregistered [ 2669.749622] Key type ._llcrypt unregistered [ 2682.969527] Key type ._llcrypt registered [ 2682.979826] Key type .llcrypt registered [ 2684.300152] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 19:44:57 (1781221497) [ 2684.863459] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2684.895108] alg: No test for adler32 (adler32-zlib) [ 2685.900206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2686.515230] LNet: 119046:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2687.414529] Key type .llcrypt unregistered [ 2687.419142] Key type ._llcrypt unregistered [ 2695.357051] Key type ._llcrypt registered [ 2695.358202] Key type .llcrypt registered [ 2695.599664] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2695.615714] alg: No test for adler32 (adler32-zlib) [ 2696.641305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2696.692147] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2706.583390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.135@tcp [ 2706.876542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2707.594939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.135@tcp [ 2707.908457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2707.966906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2708.001134] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.135@tcp [ 2708.019637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2708.071588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2708.107514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2708.182666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2708.250691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2708.314696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2708.326587] LNet: 119496:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2708.338399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2708.358476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2708.379042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2708.486182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2708.548605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2708.601519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2709.774196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2710.251418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2710.311350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2710.395384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2710.466497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2710.527837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2710.621368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2710.675043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2710.714208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2710.746158] LNet: 119614:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2710.771917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2710.818751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2710.858232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2710.959244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2711.035063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2711.142401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2712.298801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.135@tcp [ 2712.803957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2712.904772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2712.977911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2713.063396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2713.125848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2713.208450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2713.280214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2713.362349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2713.404828] LNet: 119731:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2713.425643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2713.463154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2713.494331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2713.543704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2713.634422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2713.750559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2715.002621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2715.618232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2715.682381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2715.781944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2715.840943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2715.900507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2715.971864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2716.037388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2716.128847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2716.188897] LNet: 119849:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2716.236374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2716.352309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2716.468486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2716.512621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2716.602203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2716.702312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2717.465624] LNet: 119895:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2717.476111] LNetError: 119895:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2717.509210] LNet: Removed LNI 192.168.202.35@tcp [ 2718.183762] Key type .llcrypt unregistered [ 2718.186162] Key type ._llcrypt unregistered [ 2742.790906] Key type ._llcrypt registered [ 2742.793930] Key type .llcrypt registered [ 2744.696651] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 19:45:56 (1781221556) [ 2745.207104] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2745.230946] alg: No test for adler32 (adler32-zlib) [ 2746.163818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2746.594716] LNet: 120783:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2747.739404] Key type .llcrypt unregistered [ 2747.746181] Key type ._llcrypt unregistered [ 2756.047458] Key type ._llcrypt registered [ 2756.052634] Key type .llcrypt registered [ 2756.440515] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2756.468661] alg: No test for adler32 (adler32-zlib) [ 2757.501777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2757.553881] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2767.043961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.135@tcp [ 2778.971221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2778.999867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2779.023610] LNet: Added LNI 192.168.202.35@tcp1 [8/256/0/180] [ 2780.167995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2780.736905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.135@tcp [ 2780.955529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2780.978597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2781.005798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2781.032495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2781.113464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2781.147506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2781.153501] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.135@tcp [ 2781.168965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2781.194745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2781.209670] LNet: 121436:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2781.219739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2781.233316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2781.247455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2781.268907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2781.285581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2781.306404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2781.853357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2782.007930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2782.040742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2782.072453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2782.113357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2782.142787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2782.168838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2782.195837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2782.227134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2782.238945] LNet: 121553:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2782.249490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2782.266040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2782.302168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2782.327463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2782.355117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2782.379485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2783.130255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.135@tcp [ 2783.399409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2783.431512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2783.464114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2783.491421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2783.528909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2783.575464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2783.655969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2783.718767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2783.769832] LNet: 121670:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2783.796922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2783.845797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2783.896057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2783.938980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2783.993495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2784.041866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2785.054645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2785.425377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2785.463928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2785.507557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2785.553851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2785.594074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2785.649791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2785.689381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2785.718425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2785.745920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2785.776687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2785.840055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2785.886682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2785.916368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2785.940963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2792.112836] LNet: 121869:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2792.120207] LNetError: 121869:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2792.129508] LNet: Removed LNI 192.168.202.35@tcp [ 2792.148871] LNet: Removed LNI 192.168.202.35@tcp1 [ 2792.883315] Key type .llcrypt unregistered [ 2792.891082] Key type ._llcrypt unregistered [ 2815.125918] Key type ._llcrypt registered [ 2815.127667] Key type .llcrypt registered [ 2816.437277] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 19:47:09 (1781221629) [ 2824.645398] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 19:47:17 (1781221637) [ 2833.918550] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 19:47:27 (1781221647) [ 2843.397313] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 19:47:35 (1781221655) [ 2851.827074] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 19:47:44 (1781221664) [ 2859.841974] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 19:47:52 (1781221672) [ 2867.741588] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 19:48:00 (1781221680) [ 2875.851387] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 19:48:08 (1781221688) [ 2884.020653] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 19:48:16 (1781221696) [ 2891.935702] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 19:48:24 (1781221704) [ 2900.477547] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 19:48:33 (1781221713) [ 2907.870889] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 19:48:40 (1781221720) [ 2915.940305] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 19:48:48 (1781221728) [ 2923.182898] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 19:48:56 (1781221736) [ 2932.207955] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 19:49:04 (1781221744) [ 2940.816133] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 19:49:13 (1781221753) [ 2949.359583] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 19:49:22 (1781221762) [ 2959.228259] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 19:49:31 (1781221771) [ 2959.849393] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2959.885432] alg: No test for adler32 (adler32-zlib) [ 2960.986932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2961.227230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2961.284646] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2963.491630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2963.544671] LNet: 132587:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2963.553228] LNetError: 132587:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2963.570519] LNet: Removed LNI 192.168.202.35@tcp [ 2963.611162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2963.744792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2963.814076] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2971.583438] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 19:49:44 (1781221784) [ 2980.387041] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 19:49:53 (1781221793) [ 2987.164533] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 19:50:00 (1781221800) [ 2994.620884] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 19:50:07 (1781221807) [ 3001.685349] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 19:50:14 (1781221814) [ 3010.854487] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 19:50:23 (1781221823) [ 3021.526372] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 19:50:33 (1781221833) [ 3030.694293] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 19:50:43 (1781221843) [ 3038.184328] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 19:50:51 (1781221851) [ 3046.448770] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 19:50:59 (1781221859) [ 3052.754774] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 19:51:06 (1781221866) [ 3059.918421] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 19:51:13 (1781221873) [ 3067.541188] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2764 sec ========== 19:51:20 (1781221880) [ 3068.128406] LNet: 140052:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3068.137756] LNetError: 140052:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3068.166462] LNet: Removed LNI 192.168.202.35@tcp [ 3069.119318] Key type .llcrypt unregistered [ 3069.122384] Key type ._llcrypt unregistered [ 3086.025759] Key type ._llcrypt registered [ 3086.028386] Key type .llcrypt registered [ 3086.326930] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 3086.354428] alg: No test for adler32 (adler32-zlib) [ 3087.666554] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 3088.013579] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 3089.738114] Key type lgssc registered [ 3090.821020] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3156.523304] Lustre: Mounted lustre-client [ 3162.085788] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3171.358806] Lustre: Unmounted lustre-client [ 3196.737699] Key type lgssc unregistered [ 3196.958530] LNet: 142982:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3196.962593] LNetError: 142982:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3196.976388] LNet: Removed LNI 192.168.202.35@tcp [ 3197.609181] Key type .llcrypt unregistered [ 3197.611559] Key type ._llcrypt unregistered