[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 428514056 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2421 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22BD 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 00227D (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2331 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23C1 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE23F9 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22bd-0xbffe2330] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22bc] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2331-0xbffe23c0] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23c1-0xbffe23f8] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe23f9-0xbffe2420] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.002366] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004013] kvm-guest: setup PV IPIs [ 0.007345] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009011] pid_max: default: 32768 minimum: 301 [ 0.011129] LSM: Security Framework initializing [ 0.012062] Yama: becoming mindful. [ 0.013033] SELinux: Initializing. [ 0.014063] *** VALIDATE selinux *** [ 0.022744] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026350] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027187] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028106] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029112] *** VALIDATE tmpfs *** [ 0.030467] *** VALIDATE proc *** [ 0.032005] *** VALIDATE cgroup *** [ 0.033009] *** VALIDATE cgroup2 *** [ 0.034276] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035159] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037026] Spectre V2 : User space: Vulnerable [ 0.038006] Speculative Store Bypass: Vulnerable [ 0.040820] debug: unmapping init [mem 0xffffffffad059000-0xffffffffad060fff] [ 0.042185] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043697] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044021] ... version: 2 [ 0.044870] ... bit width: 48 [ 0.045012] ... generic registers: 4 [ 0.046010] ... value mask: 0000ffffffffffff [ 0.047013] ... max period: 00007fffffffffff [ 0.048014] ... fixed-purpose events: 3 [ 0.049009] ... event mask: 000000070000000f [ 0.050379] rcu: Hierarchical SRCU implementation. [ 0.052255] smp: Bringing up secondary CPUs ... [ 0.053608] x86: Booting SMP configuration: [ 0.054025] .... node #0, CPUs: #1 #2 #3 [ 0.057487] smp: Brought up 1 node, 4 CPUs [ 0.059015] smpboot: Max logical packages: 1 [ 0.060019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.148793] node 0 deferred pages initialised in 85ms [ 0.152009] devtmpfs: initialized [ 0.153259] x86/mm: Memory block size: 128MB [ 0.155911] gcov: version magic: 0x41383552 [ 0.157306] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.158073] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.159287] pinctrl core: initialized pinctrl subsystem [ 0.160170] [ 0.160532] ************************************************************* [ 0.161017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.162012] ** ** [ 0.163014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.164012] ** ** [ 0.165010] ** This means that this kernel is built to expose internal ** [ 0.166011] ** IOMMU data structures, which may compromise security on ** [ 0.167010] ** your system. ** [ 0.168011] ** ** [ 0.169011] ** If you see this message and you are not debugging the ** [ 0.170009] ** kernel, report this immediately to your vendor! ** [ 0.171010] ** ** [ 0.172014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.173011] ************************************************************* [ 0.174599] NET: Registered protocol family 16 [ 0.175324] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.176049] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.177036] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.178408] cpuidle: using governor menu [ 0.181292] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.182488] PCI: Using configuration type 1 for base access [ 0.183118] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.189179] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.192113] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.196121] cryptd: max_cpu_qlen set to 1000 [ 0.198381] ACPI: Added _OSI(Module Device) [ 0.200018] ACPI: Added _OSI(Processor Device) [ 0.202015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.204020] ACPI: Added _OSI(Processor Aggregator Device) [ 0.208355] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.215006] ACPI: Interpreter enabled [ 0.216156] ACPI: PM: (supports S0 S3 S4 S5) [ 0.217019] ACPI: Using IOAPIC for interrupt routing [ 0.218114] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.219415] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.228454] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.229062] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.230021] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.231099] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.233448] acpiphp: Slot [2] registered [ 0.234134] acpiphp: Slot [5] registered [ 0.235156] acpiphp: Slot [6] registered [ 0.236146] acpiphp: Slot [3] registered [ 0.237108] acpiphp: Slot [4] registered [ 0.238111] acpiphp: Slot [7] registered [ 0.239186] acpiphp: Slot [8] registered [ 0.240090] acpiphp: Slot [9] registered [ 0.241161] acpiphp: Slot [10] registered [ 0.242083] acpiphp: Slot [11] registered [ 0.243191] acpiphp: Slot [12] registered [ 0.244119] acpiphp: Slot [13] registered [ 0.245107] acpiphp: Slot [14] registered [ 0.246169] acpiphp: Slot [15] registered [ 0.247073] acpiphp: Slot [16] registered [ 0.248093] acpiphp: Slot [17] registered [ 0.249120] acpiphp: Slot [18] registered [ 0.250095] acpiphp: Slot [19] registered [ 0.251127] acpiphp: Slot [20] registered [ 0.252153] acpiphp: Slot [21] registered [ 0.253086] acpiphp: Slot [22] registered [ 0.254086] acpiphp: Slot [23] registered [ 0.255111] acpiphp: Slot [24] registered [ 0.256104] acpiphp: Slot [25] registered [ 0.257140] acpiphp: Slot [26] registered [ 0.258070] acpiphp: Slot [27] registered [ 0.259065] acpiphp: Slot [28] registered [ 0.260102] acpiphp: Slot [29] registered [ 0.261139] acpiphp: Slot [30] registered [ 0.262137] acpiphp: Slot [31] registered [ 0.263068] PCI host bridge to bus 0000:00 [ 0.264019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.265021] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.266022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.267027] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.268033] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.269036] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.270184] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.272067] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.274377] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.278816] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.281045] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.282019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.283023] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.284021] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.285660] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.286778] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.287126] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.288972] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.290023] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.297017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.299019] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.303014] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.306021] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.309018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.316019] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.322041] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.331017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.340020] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.360019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.380254] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.383363] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.385497] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.387457] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.390242] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.394216] iommu: Default domain type: Passthrough [ 0.396434] SCSI subsystem initialized [ 0.398128] ACPI: bus type USB registered [ 0.399120] usbcore: registered new interface driver usbfs [ 0.401082] usbcore: registered new interface driver hub [ 0.403082] usbcore: registered new device driver usb [ 0.405181] pps_core: LinuxPPS API ver. 1 registered [ 0.407010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.410062] PTP clock support registered [ 0.412163] EDAC MC: Ver: 3.0.0 [ 0.414131] PCI: Using ACPI for IRQ routing [ 0.415797] NetLabel: Initializing [ 0.417011] NetLabel: domain hash size = 128 [ 0.419012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.421081] NetLabel: unlabeled traffic allowed by default [ 0.423151] vgaarb: loaded [ 0.424257] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.426010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.432426] clocksource: Switched to clocksource kvm-clock [ 0.547295] VFS: Disk quotas dquot_6.6.0 [ 0.548390] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.551445] *** VALIDATE ramfs *** [ 0.552852] *** VALIDATE hugetlbfs *** [ 0.553957] pnp: PnP ACPI init [ 0.555650] pnp: PnP ACPI: found 6 devices [ 0.570694] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.573068] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.574339] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.576021] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.578052] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.580347] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.582526] NET: Registered protocol family 2 [ 0.584641] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.589695] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.593052] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.598133] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.601450] TCP: Hash tables configured (established 65536 bind 65536) [ 0.604389] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.607374] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.609928] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.612655] NET: Registered protocol family 1 [ 0.616357] RPC: Registered named UNIX socket transport module. [ 0.617691] RPC: Registered udp transport module. [ 0.619261] RPC: Registered tcp transport module. [ 0.620680] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.622551] NET: Registered protocol family 44 [ 0.623730] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.625502] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.627331] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.629237] PCI: CLS 0 bytes, default 64 [ 0.630599] Unpacking initramfs... [ 2.031386] debug: unmapping init [mem 0xffff919f3cc64000-0xffff919f3ffcffff] [ 2.034287] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.035678] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.037355] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.529106] Initialise system trusted keyrings [ 2.531073] Key type blacklist registered [ 2.533114] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.543092] zbud: loaded [ 2.546588] *** VALIDATE nfs *** [ 2.547982] *** VALIDATE nfs4 *** [ 2.549892] pstore: using deflate compression [ 2.554056] Platform Keyring initialized [ 2.657279] NET: Registered protocol family 38 [ 2.658709] Key type asymmetric registered [ 2.659906] Asymmetric key parser 'x509' registered [ 2.661588] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.664258] io scheduler mq-deadline registered [ 2.665920] io scheduler kyber registered [ 2.667562] io scheduler bfq registered [ 2.670862] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.673765] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.676363] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.679032] ACPI: Power Button [PWRF] [ 2.683634] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.690248] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.701867] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.730404] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.759134] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.764240] Non-volatile memory driver v1.3 [ 2.765977] Linux agpgart interface v0.103 [ 2.800716] virtio_blk virtio1: [vda] 141800 512-byte logical blocks (72.6 MB/69.2 MiB) [ 2.803669] vda: detected capacity change from 0 to 72601600 [ 2.818994] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.821034] vdb: detected capacity change from 0 to 1073741824 [ 2.827141] libphy: Fixed MDIO Bus: probed [ 2.835511] usbcore: registered new interface driver usbserial_generic [ 2.838382] usbserial: USB Serial support registered for generic [ 2.841179] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.845545] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.847604] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.850192] mousedev: PS/2 mouse device common for all mice [ 2.852842] rtc_cmos 00:05: RTC can wake from S4 [ 2.854972] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.859478] rtc_cmos 00:05: registered as rtc0 [ 2.860444] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.861653] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.866640] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.867440] intel_pstate: CPU model not supported [ 2.871772] hid: raw HID events driver (C) Jiri Kosina [ 2.873405] usbcore: registered new interface driver usbhid [ 2.874830] usbhid: USB HID core driver [ 2.876224] drop_monitor: Initializing network drop monitor service [ 2.878561] Initializing XFRM netlink socket [ 2.880270] NET: Registered protocol family 10 [ 2.883777] Segment Routing with IPv6 [ 2.885095] NET: Registered protocol family 17 [ 2.889096] mpls_gso: MPLS GSO support [ 2.895883] RAS: Correctable Errors collector initialized. [ 2.897892] AVX version of gcm_enc/dec engaged. [ 2.899259] AES CTR mode by8 optimization enabled [ 2.972693] sched_clock: Marking stable (2972661029, 0)->(3919280269, -946619240) [ 2.975409] registered taskstats version 1 [ 2.976864] Loading compiled-in X.509 certificates [ 2.978601] zswap: loaded using pool lzo/zbud [ 2.996982] Key type big_key registered [ 3.008136] Key type encrypted registered [ 3.009630] ima: No TPM chip found, activating TPM-bypass! [ 3.011866] ima: Allocated hash algorithm: sha1 [ 3.013735] ima: No architecture policies found [ 3.015643] evm: Initialising EVM extended attributes: [ 3.017765] evm: security.selinux [ 3.019060] evm: security.ima [ 3.020197] evm: security.capability [ 3.021194] evm: HMAC attrs: 0x1 [ 3.023308] rtc_cmos 00:05: setting system clock to 2026-05-18 17:24:45 UTC (1779125085) [ 3.029903] debug: unmapping init [mem 0xffffffffae003000-0xffffffffae1fffff] [ 3.033162] debug: unmapping init [mem 0xffffffffacd82000-0xffffffffad058fff] [ 3.042098] Write protecting the kernel read-only data: 28672k [ 3.045625] debug: unmapping init [mem 0xffffffffab403000-0xffffffffab5fffff] [ 3.048852] debug: unmapping init [mem 0xffffffffabd14000-0xffffffffabdfffff] [ 3.078558] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.085623] systemd[1]: Detected virtualization kvm. [ 3.087623] systemd[1]: Detected architecture x86-64. [ 3.089075] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.114570] systemd[1]: No hostname configured. [ 3.116541] systemd[1]: Set hostname to . [ 3.119234] random: systemd: uninitialized urandom read (16 bytes read) [ 3.122143] systemd[1]: Initializing machine ID from random generator. [ 3.174119] random: ln: uninitialized urandom read (6 bytes read) [ 3.279882] random: systemd: uninitialized urandom read (16 bytes read) [ 3.283675] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.288376] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.294537] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... Starting Journal Service... Starting Setup Virtual Console... [ OK ] Reached target Sockets. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Timers. [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.000381] device-mapper: uevent: version 1.0.3 [ 4.002771] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.784946] virtio_net virtio0 ens2: renamed from eth0 [ 4.829045] random: fast init done [ 4.898684] scsi host0: ata_piix [ 4.939180] scsi host1: ata_piix [ 4.940218] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.942417] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.282418] dracut-initqueue[587]: RTNETLINK answers: File exists [ 9.525545] random: crng init done [ 9.526599] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.096188] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.348158] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.638437] SELinux: Disabled at runtime. [ 11.704808] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.713141] systemd[1]: Detected virtualization kvm. [ 11.715051] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.221147] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.225577] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.230093] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.232755] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.235228] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.243897] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.253701] systemd[1]: Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on Process Core Dump Socket. Starting Remount Root and Kernel File Systems... Starting Create list of required st…ce nodes for the current kernel... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on udev Control Socket. Starting Apply Kernel Variables... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. Mounting POSIX Message Queue File System... Mounting Huge Pages File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-getty.slice. Mounting Kernel Debug File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Journal Service. [ 12.440915] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.679666] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.961676] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.095374] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.184433] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.210471] EDAC sbridge: Ver: 1.1.2 [ 14.461240] Key type dns_resolver registered [ 14.761196] NFS: Registering the id_resolver key type [ 14.763341] Key type id_resolver registered [ 14.765026] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg230-client login: [ 41.534941] libcfs: loading out-of-tree module taints kernel. [ 41.597351] Key type ._llcrypt registered [ 41.599496] Key type .llcrypt registered [ 41.858552] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 41.868714] alg: No test for adler32 (adler32-zlib) [ 42.936037] Lustre: Lustre: Build Version: 2.17.53_24_g2ff46d7 [ 43.328585] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 44.999220] Key type lgssc registered [ 45.904491] Lustre: Echo OBD driver; http://www.lustre.org/ [ 186.237319] Lustre: Mounted lustre-client [ 191.563391] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 206.414741] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing check_logdir /tmp/testlogs/ [ 211.949453] Lustre: lustre-OST0000-osc-ffff919fa0461000: disconnect after 23s idle [ 212.556300] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing yml_node [ 217.196219] Lustre: DEBUG MARKER: Client: 2.17.53.24 [ 219.931537] Lustre: DEBUG MARKER: MDS: 2.17.53.24 [ 222.264368] Lustre: DEBUG MARKER: OSS: 2.17.53.24 [ 224.439183] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Mon May 18 13:28:25 EDT 2026 [ 242.511346] LustreError: 7970:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffff919fa0461000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 242.541129] Lustre: Unmounted lustre-client [ 274.326715] Key type lgssc unregistered [ 274.664809] LNet: 8416:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 274.669862] LNetError: 8416:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 274.691643] LNet: Removed LNI 192.168.202.30@tcp [ 275.762175] Key type .llcrypt unregistered [ 275.778568] Key type ._llcrypt unregistered [ 305.709674] Key type ._llcrypt registered [ 305.714292] Key type .llcrypt registered [ 305.946770] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 305.970390] alg: No test for adler32 (adler32-zlib) [ 306.990475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 307.078280] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 307.108044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 311.703733] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 311.789347] LNet: 9314:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 311.797664] LNetError: 9314:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 311.814606] LNet: Removed LNI 192.168.202.30@tcp [ 312.465490] Key type .llcrypt unregistered [ 312.468848] Key type ._llcrypt unregistered [ 320.507153] Key type ._llcrypt registered [ 320.509208] Key type .llcrypt registered [ 321.980577] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 13:30:02 (1779125402) [ 322.683855] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 322.705461] alg: No test for adler32 (adler32-zlib) [ 323.632767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 323.739802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 323.774214] LNet: 9737:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 323.807520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 323.848856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 323.865613] LNet: 9744:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 331.030342] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 13:30:12 (1779125412) [ 331.330589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 331.359291] LNet: 10321:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 331.373823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 331.478443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 338.106774] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 13:30:19 (1779125419) [ 338.204276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 338.248367] LNet: 10902:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 338.270362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 338.339502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 344.990923] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 13:30:26 (1779125426) [ 345.188596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 345.204321] LNet: 11495:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 345.222576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 345.282821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 352.545506] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 13:30:33 (1779125433) [ 352.710090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 352.748472] LNet: 12076:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 352.790843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 352.879955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 353.405947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 360.896994] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 13:30:41 (1779125441) [ 361.038596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 361.082910] LNet: 12663:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 361.108773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 361.186061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 361.688876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 362.358516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 370.171918] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 13:30:51 (1779125451) [ 370.331977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 370.396413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 370.529645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 376.719188] hrtimer: interrupt took 2284295 ns [ 378.484966] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 13:30:59 (1779125459) [ 378.595825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 378.632371] LNet: 13845:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 378.637358] LNet: 13845:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 378.701685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 378.827139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 378.894824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 379.557304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 379.623324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 380.346950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 380.397242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 381.009832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 381.083089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 381.733143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 381.794761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 382.475506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 382.515293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 383.282328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 383.345397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 383.802894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 383.854851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 384.564352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 384.622175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 385.298675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 385.338614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 385.919641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 385.948102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 386.429802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 386.469345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 387.114222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 387.169654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 395.674904] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 13:31:16 (1779125476) [ 396.074595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 396.111720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 396.225727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 396.291298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 403.591140] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 13:31:24 (1779125484) [ 403.898727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 403.959959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 404.024160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 404.048910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 412.010426] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 13:31:32 (1779125492) [ 412.177842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 412.197803] LNet: 15706:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 412.210684] LNet: 15706:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 412.238755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 412.350526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 412.382706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 420.382090] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 13:31:41 (1779125501) [ 420.656530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 420.731763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 420.840793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 420.926725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 429.066815] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 13:31:49 (1779125509) [ 429.197300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 429.268591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 429.393722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 429.453864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 436.303759] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 13:31:57 (1779125517) [ 436.552695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 436.579389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 436.656756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 436.696881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 444.682676] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 13:32:05 (1779125525) [ 444.916980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 444.976778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 445.164920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 445.714649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 454.001918] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 13:32:14 (1779125534) [ 454.326399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 454.432424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 454.788717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 455.171278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 464.509635] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 13:32:25 (1779125545) [ 464.581546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 464.648174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 464.747460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 465.089146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 472.818420] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 13:32:33 (1779125553) [ 473.062366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 473.136472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 473.796141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 474.141393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 482.424685] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 13:32:43 (1779125563) [ 482.556561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 482.586181] LNet: 20537:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 482.589209] LNet: 20537:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 482.616952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 482.719709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 482.781775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 490.677950] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 13:32:51 (1779125571) [ 490.759328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 490.830526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 490.932846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 490.985930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 498.314824] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 13:32:59 (1779125579) [ 498.439881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 498.497561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 498.586930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 505.899622] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 13:33:07 (1779125587) [ 506.192062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 506.236901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 506.309886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 514.591481] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 13:33:14 (1779125594) [ 514.702162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 514.754656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 514.826697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 515.580557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 522.963793] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 13:33:23 (1779125603) [ 523.232247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 523.307075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 523.372972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 523.461976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 530.245324] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 13:33:31 (1779125611) [ 530.339553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 530.403393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 530.471362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 530.503188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 536.982319] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 13:33:38 (1779125618) [ 537.271165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 537.371673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 537.489422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 537.519783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 547.172169] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 13:33:47 (1779125627) [ 547.399832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 547.540169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 547.769576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 547.863857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 558.702493] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 13:33:58 (1779125638) [ 558.957537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 559.057694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 559.169595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 559.269432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 559.297712] LNetError: 25856:0:(peer.c:2175:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 559.345270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 559.395034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 566.724556] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 13:34:07 (1779125647) [ 566.904389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 566.967348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 567.086066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.ojRk24FbpQ [ 567.228626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.ojRk24FbpQ [ 567.336111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.ojRk24FbpQ [ 575.924969] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 13:34:16 (1779125656) [ 576.126599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 576.243244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 576.392448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 576.460418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 576.551438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 583.709594] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 13:34:24 (1779125664) [ 583.809684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 583.851391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 583.944607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 586.143337] LNet: Message forwarding will be enabled in 2s [ 589.215376] LNet: Message forwarding enabled [ 589.647410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 589.657924] LNet: Message forwarding will stop in 2s to 6s [ 594.279400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 595.070512] LNet: Message forwarding disabled [ 597.428227] LNet: Message forwarding will be enabled in 2s [ 601.251940] LNet: Message forwarding enabled [ 604.943764] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 13:34:46 (1779125686) [ 605.275339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 607.279692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 607.451067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 614.231631] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 13:34:55 (1779125695) [ 614.663853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 614.700440] LNet: 28884:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 614.712405] LNet: 28884:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 13 previous similar messages [ 614.737154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 614.814477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 614.857272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 614.899762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 614.946520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 614.976922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 615.016905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 615.065525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 615.103678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 615.165443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 615.215415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 615.268272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 615.322250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 615.370920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 615.403543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 615.455276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 615.491497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 615.518431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 615.559466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 615.591933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 615.615098] LNetError: 28927:0:(router.c:624:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 622.564921] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 13:35:03 (1779125703) [ 622.687738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 622.739874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 622.864863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 630.510923] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 13:35:11 (1779125711) [ 632.825036] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg230-server [ 635.175671] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 13:35:15 (1779125715) [ 635.383795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 635.462940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 635.540894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 635.566980] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 637.650622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.202.[31-35]@tcp [ 637.693320] LNet: 30644:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 637.757102] LNetError: 30433:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.31@tcp is being used as a gateway but routing feature is not turned on [ 637.773340] LNetError: 30433:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp1 through 192.168.202.31@tcp has gone from up to down [ 638.763199] LNetError: 30433:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.35@tcp is being used as a gateway but routing feature is not turned on [ 638.777636] LNetError: 30433:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 639.786810] LNetError: 30433:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.35@tcp is being used as a gateway but routing feature is not turned on [ 639.796310] LNetError: 30433:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 641.829184] LNetError: 30433:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.32@tcp is being used as a gateway but routing feature is not turned on [ 641.836300] LNetError: 30433:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 645.457501] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 13:35:26 (1779125726) [ 645.924520] LNetError: 30433:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.32@tcp is being used as a gateway but routing feature is not turned on [ 645.950099] LNetError: 30433:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 647.639698] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg230-server [ 649.514391] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 13:35:30 (1779125730) [ 649.639358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 649.692801] LNetError: 31574:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 649.728148] LNet: Removed LNI 192.168.202.30@tcp [ 649.758215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 649.806645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 649.824576] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 650.679962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.202.[31-36/2]@tcp [ 650.717733] LNet: 31696:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 650.735968] LNet: 31696:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 650.798843] LNetError: 31583:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp103 through 192.168.202.31@tcp has gone from up to down [ 650.798964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.202.[31-36/2]@tcp [ 650.814463] LNetError: 31583:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 657.662076] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 13:35:38 (1779125738) [ 657.961314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 657.992622] LNetError: 32288:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 658.009518] LNet: Removed LNI 192.168.202.30@tcp [ 658.050460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 658.212695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 658.331916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 658.375378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 658.439963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 658.526643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 658.563276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 658.624901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 658.741700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 658.785297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 658.886891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 658.996403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 659.073988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 659.180795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 659.302739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 659.385286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 659.491731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 659.618641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 659.665745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 659.718388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 659.812816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 659.855813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 659.938378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 660.064679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 660.106552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 660.155774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 660.280078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 660.353840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 660.471268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 660.582846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 660.636553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 660.681628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 660.743809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 660.780807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 660.835549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 667.708042] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 13:35:48 (1779125748) [ 669.376545] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg230-server [ 671.633056] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 13:35:52 (1779125752) [ 673.929788] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg230-server [ 675.760590] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 13:35:56 (1779125756) [ 676.579672] Key type .llcrypt unregistered [ 676.582503] Key type ._llcrypt unregistered [ 683.845411] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 683.929942] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 683.935164] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 683.942583] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 686.738909] Key type ._llcrypt registered [ 686.745845] Key type .llcrypt registered [ 686.995277] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 687.017742] alg: No test for adler32 (adler32-zlib) [ 688.040771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 688.136465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 688.176907] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 688.965812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 688.995726] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 691.134333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 691.156948] LNet: Removed LNI 10.1.2.3@tcp [ 691.338812] LNet: 34412:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 691.344960] LNetError: 34412:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 691.373738] LNet: Removed LNI 192.168.202.30@tcp [ 691.993780] Key type .llcrypt unregistered [ 691.996431] Key type ._llcrypt unregistered [ 705.241339] Key type ._llcrypt registered [ 705.242441] Key type .llcrypt registered [ 706.592642] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 13:36:27 (1779125787) [ 707.204440] Key type .llcrypt unregistered [ 707.208206] Key type ._llcrypt unregistered [ 713.858948] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 714.067593] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 714.073034] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 714.105684] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 716.857728] Key type ._llcrypt registered [ 716.860294] Key type .llcrypt registered [ 717.032498] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 717.048806] alg: No test for adler32 (adler32-zlib) [ 717.989743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 718.048046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 718.086714] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 718.844205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 718.868171] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 722.144883] LNet: 35915:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 722.160906] LNetError: 35915:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 722.169560] LNet: Removed LNI 192.168.202.30@tcp [ 722.180400] LNet: Removed LNI 10.1.2.3@tcp [ 722.982620] Key type .llcrypt unregistered [ 722.985837] Key type ._llcrypt unregistered [ 739.312097] Key type ._llcrypt registered [ 739.314978] Key type .llcrypt registered [ 741.068309] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 13:37:01 (1779125821) [ 741.708342] Key type .llcrypt unregistered [ 741.712624] Key type ._llcrypt unregistered [ 749.407879] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 749.504532] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 749.509553] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 749.512919] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 752.572790] Key type ._llcrypt registered [ 752.582126] Key type .llcrypt registered [ 753.036291] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 753.099149] alg: No test for adler32 (adler32-zlib) [ 754.426519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 754.566211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 754.614688] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 757.156888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 757.182891] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 760.458745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 760.472821] LNet: Removed LNI 10.1.2.3@tcp [ 760.482387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 760.505366] LNet: Removed LNI 10.1.2.31@tcp [ 760.519909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 760.548608] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 761.156396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 761.207126] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 762.860923] LNet: 37842:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 762.873723] LNetError: 37842:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 762.891116] LNet: Removed LNI 10.1.2.3@tcp [ 762.952753] LNet: Removed LNI 10.1.2.31@tcp [ 764.285792] Key type .llcrypt unregistered [ 764.290805] Key type ._llcrypt unregistered [ 779.416744] Key type ._llcrypt registered [ 779.419502] Key type .llcrypt registered [ 781.089836] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 13:37:41 (1779125861) [ 781.584364] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 781.595554] alg: No test for adler32 (adler32-zlib) [ 782.598532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 782.690578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 782.725394] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 783.673937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 783.697190] LNet: 38812:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 783.705958] LNetError: 38812:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 783.726947] LNet: Removed LNI 192.168.202.30@tcp [ 783.761985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 783.838155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.202.30@tcp [ 783.863733] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 784.627724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.202.30@tcp [ 784.663855] LNet: Removed LNI 192.168.202.30@tcp [ 784.710666] LNet: 38947:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 785.461982] Key type .llcrypt unregistered [ 785.463730] Key type ._llcrypt unregistered [ 800.699502] Key type ._llcrypt registered [ 800.702840] Key type .llcrypt registered [ 802.505736] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 13:38:02 (1779125882) [ 805.063777] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg230-server [ 807.076196] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 13:38:08 (1779125888) [ 808.065469] Key type .llcrypt unregistered [ 808.073105] Key type ._llcrypt unregistered [ 814.875863] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 815.052643] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 815.068257] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 815.077930] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 817.527969] Key type ._llcrypt registered [ 817.528929] Key type .llcrypt registered [ 817.790687] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 817.848938] alg: No test for adler32 (adler32-zlib) [ 818.937550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 819.002111] LNet: Added LNI 192.168.202.30@tcp1 [8/256/0/180] [ 819.911125] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 823.076474] LNet: 40789:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 823.090569] LNetError: 40789:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 823.093419] LNet: Removed LNI 192.168.202.30@tcp1 [ 823.118737] LNet: Removed LNI 10.1.2.3@tcp2 [ 823.955728] Key type .llcrypt unregistered [ 823.960431] Key type ._llcrypt unregistered [ 838.315379] Key type ._llcrypt registered [ 838.318258] Key type .llcrypt registered [ 839.840891] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 13:38:40 (1779125920) [ 840.434452] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 840.462423] alg: No test for adler32 (adler32-zlib) [ 841.398974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 841.445936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 841.462449] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 842.372698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 842.390351] LNet: 41763:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 842.395672] LNetError: 41763:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 842.405948] LNet: Removed LNI 192.168.202.30@tcp [ 842.418664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 842.475382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 842.512958] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 850.272728] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 13:38:51 (1779125931) [ 850.482745] LNet: 42350:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 850.489372] LNetError: 42350:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 850.518483] LNet: Removed LNI 192.168.202.30@tcp [ 851.217713] Key type .llcrypt unregistered [ 851.223154] Key type ._llcrypt unregistered [ 858.187802] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 858.420979] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 858.427801] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 858.432896] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 861.023348] Key type ._llcrypt registered [ 861.030935] Key type .llcrypt registered [ 861.326315] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 861.347352] alg: No test for adler32 (adler32-zlib) [ 862.315343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 862.396500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 862.440477] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 863.168231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 863.182171] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 865.518231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 865.544722] LNet: 43100:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 865.556083] LNetError: 43100:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 865.569422] LNet: Removed LNI 192.168.202.30@tcp [ 865.577295] LNet: Removed LNI 10.1.2.3@tcp [ 865.620695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 865.710542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 865.733488] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 865.739418] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 866.262272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 866.313694] LNet: 43127:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 866.321797] LNetError: 43127:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 866.325965] LNetError: 43127:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 866.331144] LNet: Removed LNI 192.168.202.30@tcp [ 866.338809] LNet: Removed LNI 10.1.2.3@tcp [ 866.359230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 866.425166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 866.461118] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 866.468064] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 867.212875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 867.244983] LNet: Added LNI 192.168.202.30@tcp2 [8/256/0/180] [ 868.608371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 868.628753] LNet: 43289:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 868.639531] LNetError: 43289:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 868.646184] LNetError: 43289:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 868.652578] LNet: Removed LNI 192.168.202.30@tcp [ 868.654618] LNet: Removed LNI 10.1.2.3@tcp [ 868.672299] LNet: Removed LNI 192.168.202.30@tcp2 [ 868.700156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 868.785505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 868.819947] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 868.823729] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 868.829643] LNet: Added LNI 192.168.202.30@tcp2 [8/256/0/180] [ 878.017590] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 13:39:18 (1779125958) [ 878.200917] LNet: 43879:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 878.224720] LNetError: 43879:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 878.234720] LNetError: 43879:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 878.259087] LNet: Removed LNI 192.168.202.30@tcp [ 878.263541] LNet: Removed LNI 10.1.2.3@tcp [ 878.308274] LNet: Removed LNI 192.168.202.30@tcp2 [ 878.996759] Key type .llcrypt unregistered [ 878.998188] Key type ._llcrypt unregistered [ 886.451509] Key type ._llcrypt registered [ 886.455316] Key type .llcrypt registered [ 886.683540] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 886.696037] alg: No test for adler32 (adler32-zlib) [ 887.691974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 887.770182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 887.814232] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 888.759262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 888.776639] LNet: 44286:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 888.785651] LNetError: 44286:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 888.799793] LNet: Removed LNI 192.168.202.30@tcp [ 888.828417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 888.878497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 888.899538] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 889.362648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 889.396921] LNet: 44316:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 889.403800] LNetError: 44316:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 889.418633] LNet: Removed LNI 192.168.202.30@tcp [ 889.442849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 889.520613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 889.564626] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 890.154447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 890.220159] LNet: Removed LNI 192.168.202.30@tcp [ 890.301285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 890.394097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 890.412227] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 890.764891] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 890.863863] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 890.872528] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 890.875244] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 893.246789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 893.271912] LNet: 44424:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 893.279500] LNet: 44424:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 893.287355] LNetError: 44424:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 893.290103] LNetError: 44424:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 893.319509] LNet: Removed LNI 192.168.202.30@tcp [ 893.345615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 893.399606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 893.419926] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 900.165515] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 13:39:41 (1779125981) [ 900.262427] LNet: 45014:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 900.266483] LNetError: 45014:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 900.277826] LNet: Removed LNI 192.168.202.30@tcp [ 901.158118] Key type .llcrypt unregistered [ 901.163888] Key type ._llcrypt unregistered [ 908.098592] Key type ._llcrypt registered [ 908.104602] Key type .llcrypt registered [ 908.331731] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 908.349909] alg: No test for adler32 (adler32-zlib) [ 909.330780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 909.391466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 909.417752] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 910.443787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 910.464398] LNet: 45426:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 910.473038] LNetError: 45426:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 910.492061] LNet: Removed LNI 192.168.202.30@tcp [ 910.522242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 910.606449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 910.642399] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 917.823144] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 13:39:58 (1779125998) [ 918.133525] LNet: 46019:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 918.143408] LNetError: 46019:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 918.165988] LNet: Removed LNI 192.168.202.30@tcp [ 918.822956] Key type .llcrypt unregistered [ 918.827074] Key type ._llcrypt unregistered [ 927.036867] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 927.185785] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 927.204638] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 927.218602] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 929.998112] Key type ._llcrypt registered [ 929.999766] Key type .llcrypt registered [ 930.197148] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 930.214445] alg: No test for adler32 (adler32-zlib) [ 931.180298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 931.321928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 931.392817] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 932.449897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 932.493098] LNet: 46474:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 932.500267] LNetError: 46474:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 932.514576] LNet: Removed LNI 192.168.202.30@tcp [ 932.535258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 932.667693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 932.698664] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 940.959230] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 13:40:21 (1779126021) [ 941.169493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 941.201505] LNet: 47069:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 941.219195] LNetError: 47069:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 941.267167] LNet: Removed LNI 192.168.202.30@tcp [ 941.313402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 941.448860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 948.816543] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 13:40:29 (1779126029) [ 948.998838] LNet: 47648:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 949.792134] Key type .llcrypt unregistered [ 949.796641] Key type ._llcrypt unregistered [ 957.107490] Key type ._llcrypt registered [ 957.112394] Key type .llcrypt registered [ 957.451274] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 957.480930] alg: No test for adler32 (adler32-zlib) [ 958.464619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 958.516574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 958.538298] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 959.633088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 959.645888] LNet: 48057:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 959.657527] LNetError: 48057:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 959.675230] LNet: Removed LNI 192.168.202.30@tcp [ 959.732780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 959.826278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 959.855316] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 967.004293] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 13:40:48 (1779126048) [ 967.243304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 967.267939] LNet: 48651:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 967.274850] LNetError: 48651:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 967.290172] LNet: Removed LNI 192.168.202.30@tcp [ 967.307983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 967.388410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 973.641437] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 13:40:54 (1779126054) [ 975.302860] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 977.319943] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 13:40:58 (1779126058) [ 979.000137] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 980.613522] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 13:41:01 (1779126061) [ 982.307376] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 983.878623] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 13:41:05 (1779126065) [ 985.485844] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 987.254460] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 13:41:08 (1779126068) [ 988.890230] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 990.756660] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 13:41:11 (1779126071) [ 990.920921] LNet: 50695:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 991.632410] Key type .llcrypt unregistered [ 991.634792] Key type ._llcrypt unregistered [ 998.136271] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 998.349441] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 998.362916] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 998.372842] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1000.943554] Key type ._llcrypt registered [ 1000.945941] Key type .llcrypt registered [ 1001.111294] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1001.122367] alg: No test for adler32 (adler32-zlib) [ 1002.067458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1002.132972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1002.165973] LNet: Added LNI 192.168.202.30@tcp [8/512/0/180] [ 1002.194806] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1002.197540] LNet: 51028:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1002.298409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1002.330147] LNet: 51048:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1002.344353] LNetError: 51048:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1002.354344] LNet: Removed LNI 192.168.202.30@tcp [ 1002.360168] LNetError: 51048:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1002.384858] LNet: Removed LNI 10.1.2.3@tcp [ 1002.407845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1002.482995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1002.529693] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 1002.602376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1002.632429] LNet: 51073:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1002.638138] LNetError: 51073:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1002.654930] LNet: Removed LNI 192.168.202.30@tcp [ 1002.674899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1002.749800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1002.770463] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 1002.774675] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1002.777817] LNet: 51081:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1002.876988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1002.908319] LNet: 51099:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1002.913913] LNetError: 51099:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1002.919327] LNet: Removed LNI 192.168.202.30@tcp [ 1002.922893] LNetError: 51099:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1002.936967] LNet: Removed LNI 10.1.2.3@tcp [ 1002.960493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1003.081974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1003.120353] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 1003.131030] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1003.132653] LNet: 51107:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1003.301257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1003.321424] LNet: 51133:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1003.327657] LNetError: 51133:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1003.336891] LNet: Removed LNI 192.168.202.30@tcp [ 1003.341762] LNetError: 51133:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1003.361686] LNet: Removed LNI 10.1.2.3@tcp [ 1003.402887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1003.452773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1003.468796] LNet: Added LNI 192.168.202.30@tcp [8/256/0/182] [ 1003.472976] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1003.562286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1003.577327] LNet: 51161:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1003.584558] LNetError: 51161:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1003.588329] LNet: Removed LNI 192.168.202.30@tcp [ 1003.589937] LNetError: 51161:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1003.600961] LNet: Removed LNI 10.1.2.3@tcp [ 1003.623417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1003.690084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1003.710990] LNet: Added LNI 192.168.202.30@tcp [8/256/0/182] [ 1003.715105] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1003.716461] LNet: 51169:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1003.778930] LNet: Removed LNI 192.168.202.30@tcp [ 1003.788399] LNet: Removed LNI 10.1.2.3@tcp [ 1004.420688] Key type .llcrypt unregistered [ 1004.423417] Key type ._llcrypt unregistered [ 1016.871380] Key type ._llcrypt registered [ 1016.874307] Key type .llcrypt registered [ 1018.428217] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 13:41:39 (1779126099) [ 1019.025149] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1019.051826] alg: No test for adler32 (adler32-zlib) [ 1020.045873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1020.150775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1020.187122] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 1021.138848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.202.30 [ 1021.176856] LNet: 52149:0:(config.c:1628:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1021.196387] LNet: Removed LNI 192.168.202.30@tcp [ 1026.686502] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 13:41:47 (1779126107) [ 1026.839651] LNet: 52717:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1026.859707] LNetError: 52717:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1026.883881] LNet: Removed LNI 192.168.202.30@tcp [ 1027.437372] Key type .llcrypt unregistered [ 1027.440124] Key type ._llcrypt unregistered [ 1033.789915] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1034.016529] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1036.850873] Key type ._llcrypt registered [ 1036.857539] Key type .llcrypt registered [ 1037.011728] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1037.022882] alg: No test for adler32 (adler32-zlib) [ 1038.308434] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1044.350925] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 13:42:05 (1779126125) [ 1044.603181] LNet: 53638:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1044.617657] LNetError: 53638:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1044.633264] LNet: Removed LNI 10.1.2.3@tcp [ 1045.247751] Key type .llcrypt unregistered [ 1045.250190] Key type ._llcrypt unregistered [ 1051.709786] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1051.930513] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1051.935779] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1051.941889] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1054.773805] Key type ._llcrypt registered [ 1054.778250] Key type .llcrypt registered [ 1054.927340] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1054.936091] alg: No test for adler32 (adler32-zlib) [ 1055.999410] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1064.479266] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 13:42:25 (1779126145) [ 1066.079193] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1068.154273] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 13:42:29 (1779126149) [ 1068.525873] LNet: 54829:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1068.535389] LNetError: 54829:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1068.599084] LNet: Removed LNI 10.1.2.3@tcp [ 1069.479478] Key type .llcrypt unregistered [ 1069.481961] Key type ._llcrypt unregistered [ 1077.306201] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1077.524410] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1077.530610] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1077.536544] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1080.259402] Key type ._llcrypt registered [ 1080.262304] Key type .llcrypt registered [ 1080.400893] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1080.409500] alg: No test for adler32 (adler32-zlib) [ 1081.372826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1081.444533] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1090.213302] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 13:42:51 (1779126171) [ 1091.005214] LNet: 56064:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1091.011242] LNetError: 56064:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1091.030193] LNet: Removed LNI 10.1.2.3@tcp [ 1091.666402] Key type .llcrypt unregistered [ 1091.668039] Key type ._llcrypt unregistered [ 1099.088468] Key type ._llcrypt registered [ 1099.091683] Key type .llcrypt registered [ 1099.284793] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1099.297484] alg: No test for adler32 (adler32-zlib) [ 1100.327152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1100.356308] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 1108.443467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.130@tcp [ 1109.264306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.30@tcp -d 192.168.202.130@tcp -m GET -e local_interrupt -r 1 [ 1109.318352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.30@tcp -d 192.168.202.30@tcp -m GET -e local_interrupt -r 1 [ 1109.356348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.130@tcp [ 1109.376473] LNet: There was an unexpected network error while writing to 192.168.202.130: rc = -22 [ 1109.595850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1109.608313] LNet: 1 local NIs in recovery (showing 1): 192.168.202.30@tcp [ 1109.641964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1109.673336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1109.744931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1109.800158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1109.858916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1109.936380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1109.974667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1110.018059] LNet: 56506:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1110.067158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1110.161942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1110.207901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1110.254141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1110.295994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1110.326394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1110.357943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1110.962557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.30@tcp -d 192.168.202.130@tcp -m GET -e local_dropped -r 1 [ 1111.003102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.30@tcp -d 192.168.202.30@tcp -m GET -e local_dropped -r 1 [ 1111.030961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.130@tcp [ 1111.049051] LNet: There was an unexpected network error while writing to 192.168.202.130: rc = -22 [ 1111.156487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1111.172434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1111.188969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1111.208626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1111.225355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1111.248815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1111.276225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1111.305606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1111.321934] LNet: 56616:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1111.336447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1111.358770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1111.381036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1111.402319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1111.425728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1111.455640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1111.481498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1112.006570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.30@tcp -d 192.168.202.130@tcp -m GET -e local_aborted -r 1 [ 1112.044460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.30@tcp -d 192.168.202.30@tcp -m GET -e local_aborted -r 1 [ 1112.089428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.130@tcp [ 1112.104158] LNet: There was an unexpected network error while writing to 192.168.202.130: rc = -22 [ 1112.292658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1112.319439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1112.342502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1112.367608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1112.394409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1112.425561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1112.461591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1112.492582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1112.523690] LNet: 56725:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1112.566168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1112.605945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1112.660380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1112.726708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1112.796862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1112.847794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1112.910483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1113.555746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.30@tcp -d 192.168.202.130@tcp -m GET -e local_no_route -r 1 [ 1113.604878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.30@tcp -d 192.168.202.30@tcp -m GET -e local_no_route -r 1 [ 1113.666402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.130@tcp [ 1113.904530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1113.933949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1113.975521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1114.009418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1114.047474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1114.099447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1114.145170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1114.176922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1114.207506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1114.252238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1114.301549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1114.355549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1114.394701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1114.427632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1114.474806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1115.073449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.30@tcp -d 192.168.202.130@tcp -m GET -e local_timeout -r 1 [ 1115.147516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.30@tcp -d 192.168.202.30@tcp -m GET -e local_timeout -r 1 [ 1115.218454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.130@tcp [ 1115.242205] LNet: There was an unexpected network error while writing to 192.168.202.130: rc = -22 [ 1115.253945] LNet: Skipped 2 previous similar messages [ 1115.531514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1115.566721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1115.594987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1115.619718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1115.645220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1115.671498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1115.708911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1115.750790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1115.778356] LNet: 56943:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1115.787256] LNet: 56943:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1115.808408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1115.840173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1115.868947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1115.906685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1115.962825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1116.024303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1116.088915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1116.645493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.30@tcp -d 192.168.202.130@tcp -m GET -e local_error -r 1 [ 1116.675500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.30@tcp -d 192.168.202.30@tcp -m GET -e local_error -r 1 [ 1116.705277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.130@tcp [ 1116.910193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1116.939975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1116.972110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1117.011447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1117.044075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1117.073786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1117.101558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1117.132715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1117.167203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1117.189900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1117.216368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1117.241560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1117.272808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1117.309480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1117.360964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1117.611965] LNet: 57094:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1117.620504] LNetError: 57094:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1118.696539] LNet: Removed LNI 192.168.202.30@tcp [ 1119.141504] Key type .llcrypt unregistered [ 1119.147269] Key type ._llcrypt unregistered [ 1136.224228] Key type ._llcrypt registered [ 1136.226307] Key type .llcrypt registered [ 1137.464551] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 13:43:38 (1779126218) [ 1138.365764] Key type .llcrypt unregistered [ 1138.367978] Key type ._llcrypt unregistered [ 1144.824210] Key type ._llcrypt registered [ 1144.830751] Key type .llcrypt registered [ 1145.048796] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1145.062921] alg: No test for adler32 (adler32-zlib) [ 1146.006610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1146.039925] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 1152.941555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.130@tcp [ 1167.001035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1167.110504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1167.181439] LNet: Added LNI 192.168.202.30@tcp1 [8/256/0/180] [ 1169.022661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.30@tcp -d 192.168.202.130@tcp -m GET -e local_interrupt -r 1 [ 1169.100237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.30@tcp -d 192.168.202.130@tcp1 -m GET -e local_interrupt -r 1 [ 1169.173203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.30@tcp -d 192.168.202.30@tcp -m GET -e local_interrupt -r 1 [ 1169.255827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.30@tcp -d 192.168.202.30@tcp1 -m GET -e local_interrupt -r 1 [ 1169.320680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.30@tcp1 -d 192.168.202.130@tcp -m GET -e local_interrupt -r 1 [ 1169.376163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.30@tcp1 -d 192.168.202.130@tcp1 -m GET -e local_interrupt -r 1 [ 1169.414742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.30@tcp1 -d 192.168.202.30@tcp -m GET -e local_interrupt -r 1 [ 1169.448563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.30@tcp1 -d 192.168.202.30@tcp1 -m GET -e local_interrupt -r 1 [ 1169.516718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.130@tcp [ 1169.560236] LNet: There was an unexpected network error while writing to 192.168.202.130: rc = -22 [ 1169.563701] LNet: 1 local NIs in recovery (showing 1): 192.168.202.30@tcp [ 1169.953171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1169.998227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1170.032108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1170.094233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1170.136317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1170.173540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1170.207541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1170.251942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1170.282163] LNet: 58570:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1170.320366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1170.355200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1170.401464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1170.468415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1170.503021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1170.545476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1170.604769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1171.232763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.30@tcp -d 192.168.202.130@tcp -m GET -e local_dropped -r 1 [ 1171.303532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.30@tcp -d 192.168.202.130@tcp1 -m GET -e local_dropped -r 1 [ 1171.376631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.30@tcp -d 192.168.202.30@tcp -m GET -e local_dropped -r 1 [ 1171.435581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.30@tcp -d 192.168.202.30@tcp1 -m GET -e local_dropped -r 1 [ 1171.507364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.30@tcp1 -d 192.168.202.130@tcp -m GET -e local_dropped -r 1 [ 1171.574289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.30@tcp1 -d 192.168.202.130@tcp1 -m GET -e local_dropped -r 1 [ 1171.675558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.30@tcp1 -d 192.168.202.30@tcp -m GET -e local_dropped -r 1 [ 1171.745992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.30@tcp1 -d 192.168.202.30@tcp1 -m GET -e local_dropped -r 1 [ 1171.794824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.130@tcp [ 1171.832068] LNet: There was an unexpected network error while writing to 192.168.202.130: rc = -22 [ 1171.838719] LNet: Skipped 2 previous similar messages [ 1172.110224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1172.153299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1172.178606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1172.207696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1172.263550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1172.294297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1172.322431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1172.350337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1172.363988] LNet: 58692:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1172.380572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1172.407692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1172.429454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1172.488312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1172.574897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1172.621752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1172.667370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1173.301223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.30@tcp -d 192.168.202.130@tcp -m GET -e local_aborted -r 1 [ 1173.338436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.30@tcp -d 192.168.202.130@tcp1 -m GET -e local_aborted -r 1 [ 1173.383964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.30@tcp -d 192.168.202.30@tcp -m GET -e local_aborted -r 1 [ 1173.414646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.30@tcp -d 192.168.202.30@tcp1 -m GET -e local_aborted -r 1 [ 1173.438664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.30@tcp1 -d 192.168.202.130@tcp -m GET -e local_aborted -r 1 [ 1173.470185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.30@tcp1 -d 192.168.202.130@tcp1 -m GET -e local_aborted -r 1 [ 1173.508847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.30@tcp1 -d 192.168.202.30@tcp -m GET -e local_aborted -r 1 [ 1173.543666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.30@tcp1 -d 192.168.202.30@tcp1 -m GET -e local_aborted -r 1 [ 1173.604859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.130@tcp [ 1173.642742] LNet: There was an unexpected network error while writing to 192.168.202.130: rc = -22 [ 1173.653605] LNet: Skipped 2 previous similar messages [ 1174.064113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1174.103857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1174.168720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1174.231207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1174.297372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1174.367793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1174.421455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1174.475929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1174.527671] LNet: 58813:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1174.568512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1174.682993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1174.771502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1174.843575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1174.881404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1174.913733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1174.994334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1175.610669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.30@tcp -d 192.168.202.130@tcp -m GET -e local_no_route -r 1 [ 1175.652706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.30@tcp -d 192.168.202.130@tcp1 -m GET -e local_no_route -r 1 [ 1175.706601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.30@tcp -d 192.168.202.30@tcp -m GET -e local_no_route -r 1 [ 1175.764803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.30@tcp -d 192.168.202.30@tcp1 -m GET -e local_no_route -r 1 [ 1175.832856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.30@tcp1 -d 192.168.202.130@tcp -m GET -e local_no_route -r 1 [ 1175.918118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.30@tcp1 -d 192.168.202.130@tcp1 -m GET -e local_no_route -r 1 [ 1176.001502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.30@tcp1 -d 192.168.202.30@tcp -m GET -e local_no_route -r 1 [ 1176.052399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.30@tcp1 -d 192.168.202.30@tcp1 -m GET -e local_no_route -r 1 [ 1176.099124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.130@tcp [ 1176.137877] LNet: There was an unexpected network error while writing to 192.168.202.130: rc = -22 [ 1176.145979] LNet: Skipped 4 previous similar messages [ 1176.393615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1176.421785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1176.452703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1176.474758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1176.516536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1176.547146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1176.599133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1176.647481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1176.669915] LNet: 58935:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1176.692899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1176.732524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1176.780805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1176.836364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1176.866834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1176.895968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1176.944836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1177.695482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.30@tcp -d 192.168.202.130@tcp -m GET -e local_timeout -r 1 [ 1177.746726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.30@tcp -d 192.168.202.130@tcp1 -m GET -e local_timeout -r 1 [ 1177.786096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.30@tcp -d 192.168.202.30@tcp -m GET -e local_timeout -r 1 [ 1177.825597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.30@tcp -d 192.168.202.30@tcp1 -m GET -e local_timeout -r 1 [ 1177.868805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.30@tcp1 -d 192.168.202.130@tcp -m GET -e local_timeout -r 1 [ 1177.898654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.30@tcp1 -d 192.168.202.130@tcp1 -m GET -e local_timeout -r 1 [ 1177.928960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.30@tcp1 -d 192.168.202.30@tcp -m GET -e local_timeout -r 1 [ 1177.962366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.30@tcp1 -d 192.168.202.30@tcp1 -m GET -e local_timeout -r 1 [ 1178.013435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.130@tcp [ 1178.281705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1178.325773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1178.354638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1178.388483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1178.427727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1178.456194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1178.487201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1178.519677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1178.558675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1178.602805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1178.637340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1178.665934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1178.695734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1178.728473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1178.760346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1179.285649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.30@tcp -d 192.168.202.130@tcp -m GET -e local_error -r 1 [ 1179.322369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.30@tcp -d 192.168.202.130@tcp1 -m GET -e local_error -r 1 [ 1179.364563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.30@tcp -d 192.168.202.30@tcp -m GET -e local_error -r 1 [ 1179.399460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.30@tcp -d 192.168.202.30@tcp1 -m GET -e local_error -r 1 [ 1179.423791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.30@tcp1 -d 192.168.202.130@tcp -m GET -e local_error -r 1 [ 1179.444756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.30@tcp1 -d 192.168.202.130@tcp1 -m GET -e local_error -r 1 [ 1179.466906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.30@tcp1 -d 192.168.202.30@tcp -m GET -e local_error -r 1 [ 1179.496424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.30@tcp1 -d 192.168.202.30@tcp1 -m GET -e local_error -r 1 [ 1179.522089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.130@tcp [ 1179.823697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1179.853637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1179.896422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1179.933540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1179.967660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1180.016785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1180.073723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1180.108482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1180.146242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1180.184408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1180.216139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1180.251827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1180.282540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1180.310191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1180.336872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1183.847539] LNet: 59253:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1183.850962] LNetError: 59253:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1183.857258] LNet: Removed LNI 192.168.202.30@tcp [ 1184.870954] LNet: Removed LNI 192.168.202.30@tcp1 [ 1185.228755] Key type .llcrypt unregistered [ 1185.230354] Key type ._llcrypt unregistered [ 1204.103477] Key type ._llcrypt registered [ 1204.106690] Key type .llcrypt registered [ 1205.477428] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 13:44:46 (1779126286) [ 1206.414575] Key type .llcrypt unregistered [ 1206.417641] Key type ._llcrypt unregistered [ 1213.127355] Key type ._llcrypt registered [ 1213.134138] Key type .llcrypt registered [ 1213.325233] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1213.353303] alg: No test for adler32 (adler32-zlib) [ 1214.308547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1214.340316] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 1221.958308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.130@tcp [ 1222.768397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.30@tcp -d 192.168.202.130@tcp -m GET -e remote_dropped -r 1 [ 1222.813399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.30@tcp -d 192.168.202.30@tcp -m GET -e remote_dropped -r 1 [ 1222.883815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.130@tcp [ 1222.939489] LNet: There was an unexpected network error while writing to 192.168.202.130: rc = -22 [ 1223.227864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1223.263872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1223.293939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1223.331421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1223.361191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1223.387962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1223.417619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1223.452381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1223.471165] LNet: 60514:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1223.488553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1223.525209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1223.564193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1223.595576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1223.626946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1223.673980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1223.717795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1224.255988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.30@tcp -d 192.168.202.130@tcp -m GET -e remote_error -r 1 [ 1224.286839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.30@tcp -d 192.168.202.30@tcp -m GET -e remote_error -r 1 [ 1224.327530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.130@tcp [ 1224.350442] LNet: There was an unexpected network error while writing to 192.168.202.130: rc = -22 [ 1224.603184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1224.673730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1224.723966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1224.794549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1224.857494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1224.908768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1225.000761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1225.072288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1225.113038] LNet: 60623:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1225.143425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1225.217752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1225.287898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1225.367159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1225.411771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1225.447460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1225.510399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1226.218266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.30@tcp -d 192.168.202.130@tcp -m GET -e remote_timeout -r 1 [ 1226.252939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.30@tcp -d 192.168.202.30@tcp -m GET -e remote_timeout -r 1 [ 1226.297367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.130@tcp [ 1226.314814] LNet: There was an unexpected network error while writing to 192.168.202.130: rc = -22 [ 1226.534350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1226.559460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1226.588384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1226.629871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1226.662153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1226.704796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1226.751577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1226.804194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1226.831849] LNet: 60732:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1226.848427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1226.883181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1226.932952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1226.971083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1227.009403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1227.062960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1227.132377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1227.364600] LNet: 60773:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1227.370845] LNetError: 60773:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1227.387267] LNet: Removed LNI 192.168.202.30@tcp [ 1227.707457] Key type .llcrypt unregistered [ 1227.709889] Key type ._llcrypt unregistered [ 1246.247373] Key type ._llcrypt registered [ 1246.249591] Key type .llcrypt registered [ 1247.486723] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 13:45:28 (1779126328) [ 1248.596418] Key type .llcrypt unregistered [ 1248.600123] Key type ._llcrypt unregistered [ 1254.997892] Key type ._llcrypt registered [ 1255.000531] Key type .llcrypt registered [ 1255.223105] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1255.252545] alg: No test for adler32 (adler32-zlib) [ 1256.266178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1256.323356] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 1264.538830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.130@tcp [ 1273.440957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1273.487384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1273.513772] LNet: Added LNI 192.168.202.30@tcp1 [8/256/0/180] [ 1275.055315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.30@tcp -d 192.168.202.130@tcp -m GET -e remote_dropped -r 1 [ 1275.082702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.30@tcp -d 192.168.202.130@tcp1 -m GET -e remote_dropped -r 1 [ 1275.113304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.30@tcp -d 192.168.202.30@tcp -m GET -e remote_dropped -r 1 [ 1275.154559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.30@tcp -d 192.168.202.30@tcp1 -m GET -e remote_dropped -r 1 [ 1275.183834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.30@tcp1 -d 192.168.202.130@tcp -m GET -e remote_dropped -r 1 [ 1275.215622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.30@tcp1 -d 192.168.202.130@tcp1 -m GET -e remote_dropped -r 1 [ 1275.245311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.30@tcp1 -d 192.168.202.30@tcp -m GET -e remote_dropped -r 1 [ 1275.269429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.30@tcp1 -d 192.168.202.30@tcp1 -m GET -e remote_dropped -r 1 [ 1275.315721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.130@tcp [ 1275.331537] LNet: There was an unexpected network error while writing to 192.168.202.130: rc = -22 [ 1275.339546] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.130@tcp [ 1275.593820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1275.629754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1275.667816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1275.693345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1275.721935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1275.768023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1275.799271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1275.839918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1275.861594] LNet: 62250:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1275.884605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1275.936437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1275.973368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1276.008604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1276.051533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1276.102614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1276.137058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1276.194616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1276.723724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.30@tcp -d 192.168.202.130@tcp -m GET -e remote_error -r 1 [ 1276.764428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.30@tcp -d 192.168.202.130@tcp1 -m GET -e remote_error -r 1 [ 1276.796928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.30@tcp -d 192.168.202.30@tcp -m GET -e remote_error -r 1 [ 1276.831851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.30@tcp -d 192.168.202.30@tcp1 -m GET -e remote_error -r 1 [ 1276.867160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.30@tcp1 -d 192.168.202.130@tcp -m GET -e remote_error -r 1 [ 1276.907137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.30@tcp1 -d 192.168.202.130@tcp1 -m GET -e remote_error -r 1 [ 1276.943830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.30@tcp1 -d 192.168.202.30@tcp -m GET -e remote_error -r 1 [ 1276.984691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.30@tcp1 -d 192.168.202.30@tcp1 -m GET -e remote_error -r 1 [ 1277.026459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.130@tcp [ 1277.042899] LNet: There was an unexpected network error while writing to 192.168.202.130: rc = -22 [ 1277.052100] LNet: Skipped 2 previous similar messages [ 1277.316441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1277.336677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1277.369888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1277.395464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1277.422754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1277.447992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1277.478811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1277.502542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1277.524607] LNet: 62373:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1277.539639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1277.572474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1277.604557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1277.634708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1277.678400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1277.717693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1277.755420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1277.801034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1278.323678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.30@tcp -d 192.168.202.130@tcp -m GET -e remote_timeout -r 1 [ 1278.359598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.30@tcp -d 192.168.202.130@tcp1 -m GET -e remote_timeout -r 1 [ 1278.408824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.30@tcp -d 192.168.202.30@tcp -m GET -e remote_timeout -r 1 [ 1278.447309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.30@tcp -d 192.168.202.30@tcp1 -m GET -e remote_timeout -r 1 [ 1278.480080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.30@tcp1 -d 192.168.202.130@tcp -m GET -e remote_timeout -r 1 [ 1278.536473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.30@tcp1 -d 192.168.202.130@tcp1 -m GET -e remote_timeout -r 1 [ 1278.578522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.30@tcp1 -d 192.168.202.30@tcp -m GET -e remote_timeout -r 1 [ 1278.623872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.30@tcp1 -d 192.168.202.30@tcp1 -m GET -e remote_timeout -r 1 [ 1278.664161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.130@tcp [ 1278.690459] LNet: There was an unexpected network error while writing to 192.168.202.130: rc = -22 [ 1278.935310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1278.957804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1278.989255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1279.022974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1279.050462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1279.084491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1279.119158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1279.155667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1279.176460] LNet: 62497:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1279.200350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1279.227658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1279.257968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1279.294879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1279.325773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1279.371248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1279.414476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1279.443922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1284.836578] LNet: 62575:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1284.842340] LNetError: 62575:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1284.855860] LNet: Removed LNI 192.168.202.30@tcp [ 1284.885747] LNet: Removed LNI 192.168.202.30@tcp1 [ 1285.523873] Key type .llcrypt unregistered [ 1285.526627] Key type ._llcrypt unregistered [ 1304.497456] Key type ._llcrypt registered [ 1304.503649] Key type .llcrypt registered [ 1306.205582] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 13:46:26 (1779126386) [ 1308.270611] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1310.359730] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 13:46:31 (1779126391) [ 1311.385559] Key type .llcrypt unregistered [ 1311.389315] Key type ._llcrypt unregistered [ 1318.457942] Key type ._llcrypt registered [ 1318.462720] Key type .llcrypt registered [ 1318.742536] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1318.770496] alg: No test for adler32 (adler32-zlib) [ 1319.708047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1319.755521] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 1327.233546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.130@tcp [ 1328.120212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.30@tcp -d 192.168.202.130@tcp -m GET -e network_timeout -r 1 [ 1328.147554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.30@tcp -d 192.168.202.30@tcp -m GET -e network_timeout -r 1 [ 1328.177531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.130@tcp [ 1328.195175] LNet: There was an unexpected network error while writing to 192.168.202.130: rc = -22 [ 1328.394344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1328.429292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1328.487043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1328.525267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1328.580296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1328.606890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1328.675943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1328.716702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1328.746866] LNet: 64143:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1328.770598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1328.809150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1328.848150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1328.887636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1328.929170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1328.961950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1329.257650] LNet: 64182:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1329.264639] LNetError: 64182:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1329.279915] LNet: Removed LNI 192.168.202.30@tcp [ 1329.769662] Key type .llcrypt unregistered [ 1329.772840] Key type ._llcrypt unregistered [ 1348.833347] Key type ._llcrypt registered [ 1348.845948] Key type .llcrypt registered [ 1349.065242] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1349.090160] alg: No test for adler32 (adler32-zlib) [ 1350.013062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1350.046044] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 1357.684198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.130@tcp [ 1368.050629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1368.105846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1368.144954] LNet: Added LNI 192.168.202.30@tcp1 [8/256/0/180] [ 1370.206599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.30@tcp -d 192.168.202.130@tcp -m GET -e network_timeout -r 1 [ 1370.226420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.30@tcp -d 192.168.202.130@tcp1 -m GET -e network_timeout -r 1 [ 1370.256611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.30@tcp -d 192.168.202.30@tcp -m GET -e network_timeout -r 1 [ 1370.282910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.30@tcp -d 192.168.202.30@tcp1 -m GET -e network_timeout -r 1 [ 1370.309406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.30@tcp1 -d 192.168.202.130@tcp -m GET -e network_timeout -r 1 [ 1370.338719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.30@tcp1 -d 192.168.202.130@tcp1 -m GET -e network_timeout -r 1 [ 1370.365775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.30@tcp1 -d 192.168.202.30@tcp -m GET -e network_timeout -r 1 [ 1370.386197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.30@tcp1 -d 192.168.202.30@tcp1 -m GET -e network_timeout -r 1 [ 1370.411436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.130@tcp [ 1370.426204] LNet: There was an unexpected network error while writing to 192.168.202.130: rc = -22 [ 1370.536097] LNet: 1 local NIs in recovery (showing 1): 192.168.202.30@tcp [ 1370.542371] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.130@tcp [ 1370.674911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1370.733958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1370.801618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1370.876077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1370.945637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1371.011707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1371.078927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1371.135633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1371.176776] LNet: 65072:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1371.232774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1371.328101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1371.387293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1371.471939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1371.554814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1371.616259] LNet: There was an unexpected network error while writing to 192.168.202.30: rc = -22 [ 1371.620190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1376.627460] LNet: There was an unexpected network error while writing to 192.168.202.130: rc = -22 [ 1377.218752] LNet: 65146:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1377.237059] LNetError: 65146:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1377.248842] LNet: Removed LNI 192.168.202.30@tcp [ 1377.260186] LNet: Removed LNI 192.168.202.30@tcp1 [ 1377.813679] Key type .llcrypt unregistered [ 1377.819086] Key type ._llcrypt unregistered [ 1394.277344] Key type ._llcrypt registered [ 1394.279328] Key type .llcrypt registered [ 1395.629602] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 13:47:56 (1779126476) [ 1396.068283] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1396.091611] alg: No test for adler32 (adler32-zlib) [ 1397.036515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1397.085483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1397.105880] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 1397.936917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1397.964519] LNet: Added LNI 192.168.202.30@tcp1 [8/256/0/180] [ 1398.700619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.30@tcp [ 1398.797810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1398.845814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1398.901978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1398.945438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1398.996314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1399.023663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.30@tcp [ 1399.041715] LNetError: 66251:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.202.30@tcp to recovery [ 1399.135340] LNet: 1 local NIs in recovery (showing 1): 192.168.202.30@tcp [ 1399.141256] LNet: There was an unexpected network error while writing to 192.168.202.30: rc = -22 [ 1401.194624] LNet: There was an unexpected network error while writing to 192.168.202.30: rc = -22 [ 1401.538873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1405.279606] LNet: There was an unexpected network error while writing to 192.168.202.30: rc = -22 [ 1406.400599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1406.479865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1406.576936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1406.598408] LNet: 66358:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1406.610625] LNetError: 66358:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1406.621367] LNet: Removed LNI 192.168.202.30@tcp [ 1406.634172] LNet: Removed LNI 192.168.202.30@tcp1 [ 1406.690414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1406.819761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1406.859290] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 1407.629616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1407.649350] LNet: Added LNI 192.168.202.30@tcp1 [8/256/0/180] [ 1408.498798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.30@tcp [ 1408.525378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1408.583382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1408.632350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1408.678162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1408.742675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1408.791622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1408.830687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.30@tcp [ 1408.865044] LNetError: 66631:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.202.30@tcp to recovery [ 1409.824226] LNet: There was an unexpected network error while writing to 192.168.202.30: rc = -22 [ 1412.409519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1415.009720] LNet: There was an unexpected network error while writing to 192.168.202.30: rc = -22 [ 1415.024432] LNet: Skipped 1 previous similar message [ 1419.377572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1419.414518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1419.444150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1426.098913] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 13:48:27 (1779126507) [ 1426.251627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1426.274665] LNet: 67336:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1426.279391] LNetError: 67336:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1426.283692] LNetError: 67336:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1426.287456] LNet: Removed LNI 192.168.202.30@tcp [ 1426.304589] LNet: Removed LNI 192.168.202.30@tcp1 [ 1426.342978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1426.404841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1426.422259] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 1427.318420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1427.344189] LNet: Added LNI 192.168.202.30@tcp1 [8/256/0/180] [ 1428.043607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.30@tcp [ 1428.113209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1428.139942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1428.170029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1428.209421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1428.237221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1428.268425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.30@tcp --health 0 [ 1428.357839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1428.447165] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.30@tcp [ 1429.478651] LNet: There was an unexpected network error while writing to 192.168.202.30: rc = -22 [ 1429.501094] LNet: Skipped 1 previous similar message [ 1444.085741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1444.252816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1444.299440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1444.327744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.30@tcp --health 500 [ 1444.456837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1451.209828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1451.303448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1451.324200] LNet: 67831:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1451.336655] LNetError: 67831:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1451.352040] LNetError: 67831:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1451.359497] LNet: Removed LNI 192.168.202.30@tcp [ 1451.377398] LNet: Removed LNI 192.168.202.30@tcp1 [ 1451.425488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1451.491147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1451.521453] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 1452.326187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1452.344975] LNet: Added LNI 192.168.202.30@tcp1 [8/256/0/180] [ 1453.460437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.30@tcp [ 1453.545689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1453.640414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1453.679835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1453.755877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1453.839673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1453.890584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1453.941888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.30@tcp --health 0 [ 1454.563443] LNet: There was an unexpected network error while writing to 192.168.202.30: rc = -22 [ 1454.581029] LNet: Skipped 3 previous similar messages [ 1455.408558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1465.457250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1465.527220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1465.619379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1474.887467] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 13:49:15 (1779126555) [ 1475.711748] LNet: 68866:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1475.731895] LNetError: 68866:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1475.760214] LNetError: 68866:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1475.770224] LNet: Removed LNI 192.168.202.30@tcp [ 1475.799556] LNet: Removed LNI 192.168.202.30@tcp1 [ 1476.628928] Key type .llcrypt unregistered [ 1476.633042] Key type ._llcrypt unregistered [ 1484.008025] Key type ._llcrypt registered [ 1484.010774] Key type .llcrypt registered [ 1484.221995] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1484.237875] alg: No test for adler32 (adler32-zlib) [ 1485.188689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1485.224034] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 1495.319304] Lustre: DEBUG MARKER: Initial discovery [ 1495.369826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.130@tcp [ 1497.946576] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1498.044925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1498.086890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.130@tcp [ 1498.131095] LNet: There was an unexpected network error while writing to 192.168.202.130: rc = -22 [ 1498.171982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1498.208594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.202.130@tcp -r 1 -m GET -l 3 [ 1498.248609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.202.130@tcp -r 1 -m GET -e local_error [ 1498.281634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.202.130@tcp -r 1 -m PUT -l 6 [ 1498.528250] LNet: 1 local NIs in recovery (showing 1): 192.168.202.30@tcp [ 1499.777841] Lustre: DEBUG MARKER: Force oleg230-client.virtnet to discover 192.168.202.130@tcp (in background) [ 1499.887404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.130@tcp [ 1501.412791] Lustre: DEBUG MARKER: Force oleg230-server to discover 192.168.202.30@tcp [ 1503.012699] LNet: There was an unexpected network error while writing to 192.168.202.130: rc = -22 [ 1503.732478] Lustre: DEBUG MARKER: Wait for 69628 [ 1505.407689] Lustre: DEBUG MARKER: Finished wait on 69628 [ 1505.515516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1505.570465] LNet: There was an unexpected network error while writing to 192.168.202.130: rc = -22 [ 1505.583558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1505.923424] LNet: 70083:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1505.926470] LNetError: 70083:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1505.937430] LNet: Removed LNI 192.168.202.30@tcp [ 1506.408898] Key type .llcrypt unregistered [ 1506.414309] Key type ._llcrypt unregistered [ 1524.530532] Key type ._llcrypt registered [ 1524.532731] Key type .llcrypt registered [ 1525.963520] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 13:50:06 (1779126606) [ 1526.541471] Key type .llcrypt unregistered [ 1526.544494] Key type ._llcrypt unregistered [ 1532.917877] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1533.032468] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1533.035460] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1533.040168] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1535.476211] Key type ._llcrypt registered [ 1535.478412] Key type .llcrypt registered [ 1535.641974] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1535.651865] alg: No test for adler32 (adler32-zlib) [ 1536.622602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1536.740188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1536.776959] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 1537.459891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1537.479943] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1545.658749] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 13:50:26 (1779126626) [ 1546.027460] LNet: 72171:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1546.031956] LNetError: 72171:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1546.044415] LNet: Removed LNI 192.168.202.30@tcp [ 1546.064208] LNet: Removed LNI 10.1.2.3@tcp [ 1546.735380] Key type .llcrypt unregistered [ 1546.737693] Key type ._llcrypt unregistered [ 1553.997695] Key type ._llcrypt registered [ 1554.000109] Key type .llcrypt registered [ 1554.270883] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1554.285428] alg: No test for adler32 (adler32-zlib) [ 1555.189852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1555.235457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1555.253701] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 1556.173778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1556.215305] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1557.485686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.202.30@tcp [ 1558.559640] LNetError: Unexpected error -22 connecting to 192.168.202.30@tcp at host 192.168.202.30:988 [ 1565.123636] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 13:50:45 (1779126645) [ 1565.266748] LNet: 73303:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1565.273288] LNetError: 73303:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1565.285122] LNet: Removed LNI 192.168.202.30@tcp [ 1565.302050] LNet: Removed LNI 10.1.2.3@tcp [ 1565.996429] Key type .llcrypt unregistered [ 1565.998684] Key type ._llcrypt unregistered [ 1573.342481] Key type ._llcrypt registered [ 1573.351571] Key type .llcrypt registered [ 1573.572681] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1573.618563] alg: No test for adler32 (adler32-zlib) [ 1574.613451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1574.724721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1574.777216] LNet: Added LNI 192.168.202.30@tcp1 [8/256/0/180] [ 1575.553987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1575.572605] LNet: Added LNI 192.168.202.30@tcp2 [8/256/0/180] [ 1576.341173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.202.30@tcp1 --nid 192.168.202.30@tcp2 [ 1580.108359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.30@tcp1 192.168.202.30@tcp2 [ 1580.131769] LNetError: 74147:0:(lib-move.c:2007:lnet_handle_find_routed_path()) Unable to send message from 192.168.202.30@tcp1 to 192.168.202.30@tcp2 - Route table may be misconfigured [ 1580.161469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.30@tcp2 192.168.202.30@tcp1 [ 1585.991802] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 13:51:07 (1779126667) [ 1586.094036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1586.128143] LNet: 74712:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1586.143977] LNetError: 74712:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1586.152694] LNet: Removed LNI 192.168.202.30@tcp1 [ 1587.174421] LNet: Removed LNI 192.168.202.30@tcp2 [ 1587.216815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1587.266453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1587.285624] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 1588.098479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1588.124464] LNet: Added LNI 192.168.202.30@tcp1 [8/256/0/180] [ 1589.067254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.30@tcp [ 1589.150853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.30@tcp 192.168.202.30@tcp [ 1589.213804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.30@tcp1 192.168.202.30@tcp1 [ 1589.278760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.30@tcp -d 192.168.202.30@tcp -e network_timeout [ 1589.331472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.30@tcp -d 192.168.202.30@tcp1 -e network_timeout [ 1589.417624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.30@tcp1 -d 192.168.202.30@tcp -e network_timeout [ 1589.497786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.30@tcp1 -d 192.168.202.30@tcp1 -e network_timeout [ 1589.547338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.30@tcp [ 1589.586210] LNet: There was an unexpected network error while writing to 192.168.202.30: rc = -22 [ 1589.610367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1589.719442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1589.808695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1590.304436] LNet: 1 local NIs in recovery (showing 1): 192.168.202.30@tcp [ 1596.866568] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 13:51:17 (1779126677) [ 1596.962219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1596.983229] LNet: 75579:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1596.998427] LNetError: 75579:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1597.004543] LNetError: 75579:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1597.010319] LNet: Removed LNI 192.168.202.30@tcp [ 1598.054679] LNet: Removed LNI 192.168.202.30@tcp1 [ 1598.142973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1598.339084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1598.606379] LNet: 75622:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1599.024133] Key type .llcrypt unregistered [ 1599.032498] Key type ._llcrypt unregistered [ 1610.997988] Key type ._llcrypt registered [ 1611.000300] Key type .llcrypt registered [ 1612.212368] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 13:51:33 (1779126693) [ 1612.686536] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1612.698900] alg: No test for adler32 (adler32-zlib) [ 1613.628966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1615.240690] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1617.018884] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 13:51:38 (1779126698) [ 1617.124865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1617.150897] LNet: 76717:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1617.180526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1617.258547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1617.297093] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 1618.356915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1618.390213] LNet: Added LNI 192.168.202.30@tcp1 [8/256/0/180] [ 1619.485567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.30@tcp [ 1619.518507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.30@tcp1 [ 1619.549496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.30@tcp1 [ 1627.159831] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 13:51:48 (1779126708) [ 1629.555441] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg230-server [ 1631.248821] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 13:51:52 (1779126712) [ 1633.472617] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg230-server [ 1635.409646] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 13:51:56 (1779126716) [ 1637.544849] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg230-server [ 1639.188373] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 13:52:00 (1779126720) [ 1641.293355] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg230-server [ 1643.287659] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 13:52:04 (1779126724) [ 1645.707575] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg230-server [ 1647.239021] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 13:52:08 (1779126728) [ 1649.762887] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg230-server [ 1651.350492] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 13:52:12 (1779126732) [ 1654.308436] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg230-server [ 1656.584814] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 13:52:17 (1779126737) [ 1659.094504] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg230-server [ 1660.909529] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 13:52:22 (1779126742) [ 1662.762134] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg230-server [ 1664.518177] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 13:52:25 (1779126745) [ 1664.670491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1664.704334] LNet: 80597:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1664.715130] LNetError: 80597:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1664.725534] LNet: Removed LNI 192.168.202.30@tcp [ 1665.769167] LNet: Removed LNI 192.168.202.30@tcp1 [ 1665.811789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1665.867496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1665.919273] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 1666.630410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1666.764935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.30@tcp [ 1666.984870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1667.018409] LNet: 80751:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1667.027456] LNetError: 80751:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1667.034848] LNetError: 80751:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1667.048900] LNet: Removed LNI 192.168.202.30@tcp [ 1667.061363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1667.097887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1667.120487] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 1668.033903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.30@tcp [ 1668.212203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1668.238798] LNetError: 80897:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1668.275357] LNet: Removed LNI 192.168.202.30@tcp [ 1668.320589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1668.373246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1668.408782] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 1669.203382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1669.251599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.30@tcp [ 1669.498468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1669.569687] LNet: 81044:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1669.582359] LNet: 81044:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1670.650853] LNet: Removed LNI 192.168.202.30@tcp [ 1670.686266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1670.753482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1670.787414] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 1671.417718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.30@tcp [ 1671.686890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1671.709788] LNetError: 81189:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1671.728475] LNetError: 81189:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1671.751596] LNet: Removed LNI 192.168.202.30@tcp [ 1671.858810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1671.958239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1671.999313] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 1672.778701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1672.846148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.30@tcp [ 1673.069394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1674.159566] LNet: Removed LNI 192.168.202.30@tcp [ 1674.197404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1674.263906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1674.296626] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 1675.135275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.30@tcp [ 1675.572942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1675.616897] LNet: 81482:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1675.624213] LNet: 81482:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1676.713348] LNet: Removed LNI 192.168.202.30@tcp [ 1676.744465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1676.824872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1676.857527] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 1678.029916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1678.095917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.30@tcp [ 1678.360455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1678.378282] LNetError: 81629:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1678.391359] LNetError: 81629:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1678.410426] LNet: Removed LNI 192.168.202.30@tcp [ 1678.434466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1678.529318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1678.569218] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 1679.905883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.30@tcp [ 1680.297570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1681.446992] LNet: Removed LNI 192.168.202.30@tcp [ 1681.514699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1681.616841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1681.668326] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 1682.401278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1682.503854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.30@tcp [ 1682.928192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1684.018158] LNet: Removed LNI 192.168.202.30@tcp [ 1684.077984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1684.180354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1684.242750] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 1685.267182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.30@tcp [ 1685.511632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1685.552965] LNet: 82067:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1685.571306] LNet: 82067:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1686.639421] LNet: Removed LNI 192.168.202.30@tcp [ 1686.695086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1686.746979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1686.767352] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 1687.397474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1687.480718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.30@tcp [ 1687.728719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1687.752269] LNetError: 82215:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1687.756773] LNetError: 82215:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1688.806798] LNet: Removed LNI 192.168.202.30@tcp [ 1688.864888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1688.976649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1689.024855] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 1689.594517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.30@tcp [ 1689.801653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1690.859968] LNet: Removed LNI 192.168.202.30@tcp [ 1690.904971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1691.002638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1691.028908] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 1691.798774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1691.859358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1691.880821] LNet: Removed LNI 192.168.202.30@tcp [ 1691.894501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1691.948608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1691.966182] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 1692.544413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1699.327056] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 13:53:00 (1779126780) [ 1699.532524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1699.589497] LNet: Removed LNI 192.168.202.30@tcp [ 1699.602934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1699.647172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1699.670771] LNet: Added LNI 192.168.202.30@tcp231 [8/256/0/180] [ 1700.863032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1700.906335] LNet: Removed LNI 192.168.202.30@tcp231 [ 1700.932395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1701.009295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1701.029539] LNet: Added LNI 192.168.202.30@tcp231 [8/256/0/180] [ 1701.032610] LNet: 83342:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1701.347171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1701.369299] LNet: Removed LNI 192.168.202.30@tcp231 [ 1701.376615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1701.394421] LNet: Added LNI 192.168.202.30@tcp231 [8/256/0/180] [ 1702.544882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1702.562681] LNet: 83487:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1702.567554] LNet: 83487:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 1702.583370] LNet: Removed LNI 192.168.202.30@tcp231 [ 1702.600796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1702.687485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1702.723848] LNet: Added LNI 192.168.202.30@tcp231 [8/256/0/0] [ 1702.730680] LNet: 83495:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1703.205465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1703.236402] LNet: Removed LNI 192.168.202.30@tcp231 [ 1703.261919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1703.298279] LNet: Added LNI 192.168.202.30@tcp231 [8/256/0/0] [ 1704.365969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1704.400106] LNetError: 83638:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1704.406657] LNetError: 83638:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1704.434492] LNet: Removed LNI 192.168.202.30@tcp231 [ 1704.462951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1704.561573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1704.617280] LNet: Added LNI 192.168.202.30@tcp231 [8/256/0/60] [ 1704.630653] LNet: 83646:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1705.182662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1705.212307] LNet: Removed LNI 192.168.202.30@tcp231 [ 1705.235270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1705.269095] LNet: Added LNI 192.168.202.30@tcp231 [8/256/0/60] [ 1706.258856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1706.289654] LNet: Removed LNI 192.168.202.30@tcp231 [ 1706.321780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1706.421167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1706.470128] LNet: Added LNI 192.168.202.30@tcp231 [8/256/0/180] [ 1707.074577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1707.107069] LNet: Removed LNI 192.168.202.30@tcp231 [ 1707.132786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1707.164243] LNet: Added LNI 192.168.202.30@tcp231 [8/256/0/180] [ 1715.269682] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 13:53:16 (1779126796) [ 1715.386761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1715.461069] LNet: Removed LNI 192.168.202.30@tcp231 [ 1715.514404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1715.600786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1715.636741] LNet: Added LNI 192.168.202.30@tcp232 [8/256/0/180] [ 1716.442548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1716.498734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1716.544753] LNet: Removed LNI 192.168.202.30@tcp232 [ 1716.565523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1716.591759] LNet: Added LNI 192.168.202.30@tcp232 [8/256/0/180] [ 1717.495971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1717.540143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1717.581361] LNet: Removed LNI 192.168.202.30@tcp232 [ 1717.611464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1717.648380] LNet: Added LNI 192.168.202.30@tcp232 [8/256/0/180] [ 1718.502708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1718.543594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1718.599163] LNet: Removed LNI 192.168.202.30@tcp232 [ 1718.623217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1718.652792] LNet: Added LNI 192.168.202.30@tcp232 [8/256/0/180] [ 1719.386459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1719.425244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1719.462847] LNet: Removed LNI 192.168.202.30@tcp232 [ 1719.510398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1726.253350] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 13:53:27 (1779126807) [ 1727.255354] Key type .llcrypt unregistered [ 1727.259736] Key type ._llcrypt unregistered [ 1734.074719] Key type ._llcrypt registered [ 1734.077343] Key type .llcrypt registered [ 1734.343789] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1734.376774] alg: No test for adler32 (adler32-zlib) [ 1735.368216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1735.396425] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 1743.316258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.130@tcp [ 1752.133313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1752.221667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1752.253437] LNet: Added LNI 192.168.202.30@tcp1 [8/256/0/180] [ 1753.706995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.30@tcp -d 192.168.202.130@tcp -m GET -e local_interrupt -r 2 [ 1753.733520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.30@tcp -d 192.168.202.130@tcp1 -m GET -e local_interrupt -r 2 [ 1753.764329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.30@tcp -d 192.168.202.30@tcp -m GET -e local_interrupt -r 2 [ 1753.789979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.30@tcp -d 192.168.202.30@tcp1 -m GET -e local_interrupt -r 2 [ 1753.820951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.30@tcp1 -d 192.168.202.130@tcp -m GET -e local_interrupt -r 2 [ 1753.854144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.30@tcp1 -d 192.168.202.130@tcp1 -m GET -e local_interrupt -r 2 [ 1753.901602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.30@tcp1 -d 192.168.202.30@tcp -m GET -e local_interrupt -r 2 [ 1753.935883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.30@tcp1 -d 192.168.202.30@tcp1 -m GET -e local_interrupt -r 2 [ 1754.018082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.30@tcp 192.168.202.130@tcp [ 1754.054453] LNet: There was an unexpected network error while writing to 192.168.202.130: rc = -22 [ 1754.062854] LNet: 1 local NIs in recovery (showing 1): 192.168.202.30@tcp [ 1754.335935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1754.379172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1754.410895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1754.443969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1754.479547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1754.508286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1754.542396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1754.573594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1754.589146] LNet: 86303:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1754.603457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1754.630335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1754.675833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1754.730961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1754.777562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1754.813698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1755.370269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.30@tcp -d 192.168.202.130@tcp -m GET -e local_dropped -r 2 [ 1755.410700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.30@tcp -d 192.168.202.130@tcp1 -m GET -e local_dropped -r 2 [ 1755.481369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.30@tcp -d 192.168.202.30@tcp -m GET -e local_dropped -r 2 [ 1755.517832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.30@tcp -d 192.168.202.30@tcp1 -m GET -e local_dropped -r 2 [ 1755.560083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.30@tcp1 -d 192.168.202.130@tcp -m GET -e local_dropped -r 2 [ 1755.623283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.30@tcp1 -d 192.168.202.130@tcp1 -m GET -e local_dropped -r 2 [ 1755.655657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.30@tcp1 -d 192.168.202.30@tcp -m GET -e local_dropped -r 2 [ 1755.694887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.30@tcp1 -d 192.168.202.30@tcp1 -m GET -e local_dropped -r 2 [ 1755.752540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.30@tcp 192.168.202.130@tcp [ 1755.799944] LNet: There was an unexpected network error while writing to 192.168.202.130: rc = -22 [ 1756.159272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1756.198973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1756.220612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1756.250220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1756.289611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1756.310748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1756.342284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1756.358393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1756.370558] LNet: 86424:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1756.389596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1756.409183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1756.434673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1756.464596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1756.491971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1756.517662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1756.909624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.30@tcp -d 192.168.202.130@tcp -m GET -e local_aborted -r 2 [ 1756.934274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.30@tcp -d 192.168.202.130@tcp1 -m GET -e local_aborted -r 2 [ 1756.978801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.30@tcp -d 192.168.202.30@tcp -m GET -e local_aborted -r 2 [ 1757.022067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.30@tcp -d 192.168.202.30@tcp1 -m GET -e local_aborted -r 2 [ 1757.048812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.30@tcp1 -d 192.168.202.130@tcp -m GET -e local_aborted -r 2 [ 1757.072335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.30@tcp1 -d 192.168.202.130@tcp1 -m GET -e local_aborted -r 2 [ 1757.089507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.30@tcp1 -d 192.168.202.30@tcp -m GET -e local_aborted -r 2 [ 1757.107868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.30@tcp1 -d 192.168.202.30@tcp1 -m GET -e local_aborted -r 2 [ 1757.125327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.30@tcp 192.168.202.130@tcp [ 1757.135769] LNet: There was an unexpected network error while writing to 192.168.202.130: rc = -22 [ 1757.261180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1757.283219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1757.305643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1757.330199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1757.360855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1757.387512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1757.432485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1757.475261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1757.494767] LNet: 86544:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1757.523969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1757.560961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1757.591876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1757.624787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1757.658624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1757.685427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1758.164466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.30@tcp -d 192.168.202.130@tcp -m GET -e local_no_route -r 2 [ 1758.205308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.30@tcp -d 192.168.202.130@tcp1 -m GET -e local_no_route -r 2 [ 1758.255940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.30@tcp -d 192.168.202.30@tcp -m GET -e local_no_route -r 2 [ 1758.314886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.30@tcp -d 192.168.202.30@tcp1 -m GET -e local_no_route -r 2 [ 1758.352822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.30@tcp1 -d 192.168.202.130@tcp -m GET -e local_no_route -r 2 [ 1758.425627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.30@tcp1 -d 192.168.202.130@tcp1 -m GET -e local_no_route -r 2 [ 1758.513682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.30@tcp1 -d 192.168.202.30@tcp -m GET -e local_no_route -r 2 [ 1758.555349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.30@tcp1 -d 192.168.202.30@tcp1 -m GET -e local_no_route -r 2 [ 1758.577667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.30@tcp 192.168.202.130@tcp [ 1758.783918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1758.812229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1758.834973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1758.864945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1758.904654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1758.947414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1758.992901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1759.089819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1759.151654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1759.207985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1759.298634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1759.358319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1759.417529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1759.468849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1760.138796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.30@tcp -d 192.168.202.130@tcp -m GET -e local_timeout -r 2 [ 1760.184493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.30@tcp -d 192.168.202.130@tcp1 -m GET -e local_timeout -r 2 [ 1760.214345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.30@tcp -d 192.168.202.30@tcp -m GET -e local_timeout -r 2 [ 1760.240371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.30@tcp -d 192.168.202.30@tcp1 -m GET -e local_timeout -r 2 [ 1760.276541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.30@tcp1 -d 192.168.202.130@tcp -m GET -e local_timeout -r 2 [ 1760.320781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.30@tcp1 -d 192.168.202.130@tcp1 -m GET -e local_timeout -r 2 [ 1760.389465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.30@tcp1 -d 192.168.202.30@tcp -m GET -e local_timeout -r 2 [ 1760.449636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.30@tcp1 -d 192.168.202.30@tcp1 -m GET -e local_timeout -r 2 [ 1760.529961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.30@tcp 192.168.202.130@tcp [ 1760.575291] LNet: There was an unexpected network error while writing to 192.168.202.130: rc = -22 [ 1760.579946] LNet: Skipped 1 previous similar message [ 1760.974676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1761.001639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1761.047172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1761.090915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1761.147910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1761.198975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1761.242515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1761.309638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1761.336348] LNet: 86784:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1761.346612] LNet: 86784:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1761.379619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1761.427499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1761.456959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1761.516996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1761.559271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1761.618152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1762.395932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.30@tcp -d 192.168.202.130@tcp -m GET -e remote_dropped -r 2 [ 1762.460840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.30@tcp -d 192.168.202.130@tcp1 -m GET -e remote_dropped -r 2 [ 1762.514837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.30@tcp -d 192.168.202.30@tcp -m GET -e remote_dropped -r 2 [ 1762.580963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.30@tcp -d 192.168.202.30@tcp1 -m GET -e remote_dropped -r 2 [ 1762.643961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.30@tcp1 -d 192.168.202.130@tcp -m GET -e remote_dropped -r 2 [ 1762.697926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.30@tcp1 -d 192.168.202.130@tcp1 -m GET -e remote_dropped -r 2 [ 1762.746180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.30@tcp1 -d 192.168.202.30@tcp -m GET -e remote_dropped -r 2 [ 1762.814033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.30@tcp1 -d 192.168.202.30@tcp1 -m GET -e remote_dropped -r 2 [ 1762.859273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.30@tcp 192.168.202.130@tcp [ 1762.896805] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.130@tcp [ 1763.222923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1763.255260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1763.277601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1763.310988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1763.336420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1763.380312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1763.408062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1763.436882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1763.467543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1763.490460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1763.513253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1763.556832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1763.590451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1763.619494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1769.575339] LNet: 86978:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1769.582197] LNetError: 86978:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1769.591895] LNet: Removed LNI 192.168.202.30@tcp [ 1769.606973] LNet: Removed LNI 192.168.202.30@tcp1 [ 1770.121612] Key type .llcrypt unregistered [ 1770.124054] Key type ._llcrypt unregistered [ 1791.602193] Key type ._llcrypt registered [ 1791.603930] Key type .llcrypt registered [ 1793.224470] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 13:54:33 (1779126873) [ 1793.552849] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1793.562946] alg: No test for adler32 (adler32-zlib) [ 1794.496519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1794.530868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1794.542936] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 1795.140049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1795.185780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.30@tcp [ 1795.208700] LNetError: 87961:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.202.30@tcp to recovery [ 1795.298148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.202.30@tcp [ 1795.355969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1800.917691] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 13:54:42 (1779126882) [ 1801.079880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1801.120585] LNet: 88546:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1801.130860] LNetError: 88546:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1801.181602] LNet: Removed LNI 192.168.202.30@tcp [ 1801.221269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1801.266138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1801.295780] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 1802.051309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1802.074082] LNet: Added LNI 192.168.202.30@tcp1 [8/256/0/180] [ 1803.206790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.202.30@tcp [ 1803.230469] LNetError: 88816:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.202.30@tcp to recovery [ 1803.314598] LNet: 1 local NIs in recovery (showing 1): 192.168.202.30@tcp [ 1803.332371] LNet: There was an unexpected network error while writing to 192.168.202.30: rc = -22 [ 1803.472831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.202.30@tcp [ 1811.781500] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 13:54:53 (1779126893) [ 1813.688613] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg230-server [ 1815.440562] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 13:54:56 (1779126896) [ 1817.289693] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg230-server [ 1819.125170] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 13:55:00 (1779126900) [ 1821.068020] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg230-server [ 1822.426643] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 13:55:03 (1779126903) [ 1822.541277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1822.605240] LNet: 90433:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1822.618943] LNetError: 90433:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1822.627600] LNet: Removed LNI 192.168.202.30@tcp [ 1822.662617] LNet: Removed LNI 192.168.202.30@tcp1 [ 1822.690929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1822.790253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1822.810413] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 1823.529747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1823.619303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1823.672256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1823.772366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1823.820885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1823.865597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1829.923959] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 13:55:11 (1779126911) [ 1830.063270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1830.094744] LNet: 91157:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1830.100223] LNetError: 91157:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1830.103248] LNetError: 91157:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1830.124943] LNet: Removed LNI 192.168.202.30@tcp [ 1830.160962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1830.218467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1830.239343] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 1836.933114] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 13:55:18 (1779126918) [ 1837.109975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1837.131881] LNet: 91868:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1837.147342] LNetError: 91868:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1837.183804] LNet: Removed LNI 192.168.202.30@tcp [ 1837.214276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1837.265351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1837.287348] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 1844.346817] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 13:55:25 (1779126925) [ 1845.926644] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1847.608945] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 13:55:28 (1779126928) [ 1848.043464] LNet: 92903:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1848.061656] LNetError: 92903:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1848.081981] LNet: Removed LNI 192.168.202.30@tcp [ 1848.798486] Key type .llcrypt unregistered [ 1848.803867] Key type ._llcrypt unregistered [ 1856.760897] Key type ._llcrypt registered [ 1856.765424] Key type .llcrypt registered [ 1856.947291] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1856.970072] alg: No test for adler32 (adler32-zlib) [ 1857.961889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1858.020950] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 1865.012430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.130@tcp [ 1870.493197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.202.130@tcp [ 1870.952643] LNet: 93325:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1870.956655] LNetError: 93325:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1870.964505] LNet: Removed LNI 192.168.202.30@tcp [ 1871.356920] Key type .llcrypt unregistered [ 1871.360746] Key type ._llcrypt unregistered [ 1885.688263] Key type ._llcrypt registered [ 1885.692285] Key type .llcrypt registered [ 1887.353573] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1888.913645] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1890.971274] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 13:56:11 (1779126971) [ 1893.044112] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1894.908366] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 13:56:15 (1779126975) [ 1896.607590] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg230-server [ 1898.392428] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 13:56:19 (1779126979) [ 1900.433731] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg230-server [ 1901.861545] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 13:56:23 (1779126983) [ 1902.597614] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1902.622085] alg: No test for adler32 (adler32-zlib) [ 1903.813404] Lustre: Lustre: Build Version: 2.17.53_24_g2ff46d7 [ 1904.067968] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 1905.807595] Key type lgssc registered [ 1907.608679] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1933.477315] Key type lgssc unregistered [ 1933.769786] LNet: 96537:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1933.779709] LNetError: 96537:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1933.801590] LNet: Removed LNI 192.168.202.30@tcp [ 1934.689082] Key type .llcrypt unregistered [ 1934.692986] Key type ._llcrypt unregistered [ 1952.607571] Key type ._llcrypt registered [ 1952.609810] Key type .llcrypt registered [ 1953.971414] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 13:57:14 (1779127034) [ 1954.892410] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1954.910661] alg: No test for adler32 (adler32-zlib) [ 1955.836323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1955.890723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1955.922933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1955.961295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1956.009569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1963.216124] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 13:57:24 (1779127044) [ 1963.466787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1963.481549] LNet: 97973:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1963.496966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1963.582747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1963.620860] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 1964.546452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1964.586519] LNet: 98117:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1964.603641] LNetError: 98117:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1964.626725] LNet: Removed LNI 192.168.202.30@tcp [ 1964.651981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1964.740325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1964.786879] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 1965.370486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1965.402610] LNetError: 98146:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1965.428124] LNet: Removed LNI 192.168.202.30@tcp [ 1965.458596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1965.492955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1965.513667] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 1972.236915] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 13:57:33 (1779127053) [ 1972.449620] LNet: 98737:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1972.455894] LNet: 98737:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1972.462538] LNetError: 98737:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1972.479642] LNet: Removed LNI 192.168.202.30@tcp [ 1973.063515] Key type .llcrypt unregistered [ 1973.067502] Key type ._llcrypt unregistered [ 1979.967782] Key type ._llcrypt registered [ 1979.969769] Key type .llcrypt registered [ 1980.147239] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1980.165908] alg: No test for adler32 (adler32-zlib) [ 1981.162487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1981.245569] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 1981.247869] LNet: *** cfs_fail_loc=e003, val=100*** [ 1982.109149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.30@tcp [ 1982.131310] LNet: *** cfs_fail_loc=e003, val=100*** [ 1982.134160] LNet: Skipped 2 previous similar messages [ 1982.139102] Ping reply received with mapping: [ 1982.140495] found metadata reply NI[192.168.202.30@tcp] [ 1982.145241] 1e ca a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1982.148303] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1982.153340] 5a 5a 5a 5a 5a 5a 5a 5a [ 1982.396205] LNet: 99139:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1982.405559] LNetError: 99139:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1982.421783] LNet: Removed LNI 192.168.202.30@tcp [ 1983.194288] Key type .llcrypt unregistered [ 1983.201039] Key type ._llcrypt unregistered [ 1997.207370] Key type ._llcrypt registered [ 1997.209436] Key type .llcrypt registered [ 1998.549560] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 13:57:59 (1779127079) [ 1999.210380] Key type .llcrypt unregistered [ 1999.212386] Key type ._llcrypt unregistered [ 2006.783561] Key type ._llcrypt registered [ 2006.787188] Key type .llcrypt registered [ 2007.086570] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2007.131269] alg: No test for adler32 (adler32-zlib) [ 2008.326975] LNetError: 100194:0:(api-ni.c:2729:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2008.753607] Key type .llcrypt unregistered [ 2008.764196] Key type ._llcrypt unregistered [ 2013.750757] Key type ._llcrypt registered [ 2013.754279] Key type .llcrypt registered [ 2014.862245] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 13:58:16 (1779127096) [ 2015.571352] Key type .llcrypt unregistered [ 2015.573065] Key type ._llcrypt unregistered [ 2021.985488] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2021.989693] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2024.777721] Key type ._llcrypt registered [ 2024.781672] Key type .llcrypt registered [ 2025.063514] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2025.083463] alg: No test for adler32 (adler32-zlib) [ 2026.109066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2026.234383] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2036.215538] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 13:58:37 (1779127117) [ 2036.353196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2036.405983] LNet: 101976:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2036.414652] LNetError: 101976:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2036.434422] LNet: Removed LNI 10.1.2.3@tcp [ 2036.470286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2037.340681] Key type .llcrypt unregistered [ 2037.343875] Key type ._llcrypt unregistered [ 2044.699492] Key type ._llcrypt registered [ 2044.709895] Key type .llcrypt registered [ 2045.026419] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2045.051874] alg: No test for adler32 (adler32-zlib) [ 2046.053934] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 2046.149106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.30@tcp [ 2046.299422] LNet: 102277:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2046.309299] LNetError: 102277:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2047.337479] LNet: Removed LNI 192.168.202.30@tcp [ 2048.211592] Key type .llcrypt unregistered [ 2048.216578] Key type ._llcrypt unregistered [ 2060.646247] Key type ._llcrypt registered [ 2060.648431] Key type .llcrypt registered [ 2061.898035] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 13:59:02 (1779127142) [ 2062.713296] Key type .llcrypt unregistered [ 2062.718236] Key type ._llcrypt unregistered [ 2070.462375] Key type ._llcrypt registered [ 2070.472694] Key type .llcrypt registered [ 2070.869916] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2070.901028] alg: No test for adler32 (adler32-zlib) [ 2078.724841] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 13:59:19 (1779127159) [ 2078.907678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2078.935459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2078.983206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2079.003435] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 2079.826396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2079.860161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2079.918863] LNet: Removed LNI 192.168.202.30@tcp [ 2080.399524] LNet: 104030:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2080.909583] Key type .llcrypt unregistered [ 2080.912911] Key type ._llcrypt unregistered [ 2094.285981] Key type ._llcrypt registered [ 2094.289269] Key type .llcrypt registered [ 2095.374474] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 13:59:36 (1779127176) [ 2095.791132] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2095.810032] alg: No test for adler32 (adler32-zlib) [ 2096.745168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2096.780289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2096.800843] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 2097.556531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.30@tcp [ 2097.598227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.202.30@tcp [ 2097.615508] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2097.618851] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2097.622467] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2097, 0, 0 [ 2097.626967] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2097.629660] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2097.632945] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2097.637774] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2097, 0, 0 [ 2097.642333] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2097.644667] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2097.647882] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2097.651151] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2097, 0, 2147 [ 2097.655991] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2097.658492] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2097.662136] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2097.665548] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2097, 0, 2147 [ 2097.669560] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2103.932816] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 13:59:44 (1779127184) [ 2104.436726] LNet: 105574:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2104.440824] LNetError: 105574:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2104.450221] LNet: Removed LNI 192.168.202.30@tcp [ 2105.063743] Key type .llcrypt unregistered [ 2105.066876] Key type ._llcrypt unregistered [ 2110.994564] Key type ._llcrypt registered [ 2110.996078] Key type .llcrypt registered [ 2111.191156] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2111.206831] alg: No test for adler32 (adler32-zlib) [ 2112.140833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2112.185795] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 2118.876233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.130@tcp [ 2128.987639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2129.077228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2129.128108] LNet: Added LNI 192.168.202.30@tcp1 [8/256/0/180] [ 2130.250088] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2130.422423] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2130.425973] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2130.430682] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2133.034310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2133.056991] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2145.830486] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 14:00:26 (1779127226) [ 2146.180056] LNet: 107085:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2146.184966] LNetError: 107085:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2146.191418] LNet: Removed LNI 192.168.202.30@tcp [ 2147.231172] LNet: Removed LNI 192.168.202.30@tcp1 [ 2147.233757] LNetError: 107085:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2147.237566] LNetError: 107085:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2147.247981] LNet: Removed LNI 10.1.2.3@tcp99 [ 2147.863702] Key type .llcrypt unregistered [ 2147.876933] Key type ._llcrypt unregistered [ 2156.217515] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2156.232495] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2158.947626] Key type ._llcrypt registered [ 2158.953515] Key type .llcrypt registered [ 2159.112107] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2159.123196] alg: No test for adler32 (adler32-zlib) [ 2160.118163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2160.184725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2160.226834] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 2161.077562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2161.096429] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2164.297162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2164.341604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2164.380543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2164.415143] LNetError: 107833:0:(peer.c:2175:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2164.487393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2164.585611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.30@tcp [ 2165.387632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2165.471333] LNet: 107854:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2165.479702] LNetError: 107854:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2165.488309] LNet: Removed LNI 192.168.202.30@tcp [ 2165.497059] LNet: Removed LNI 10.1.2.3@tcp [ 2166.242706] Key type .llcrypt unregistered [ 2166.245497] Key type ._llcrypt unregistered [ 2180.315596] Key type ._llcrypt registered [ 2180.320537] Key type .llcrypt registered [ 2181.900280] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 14:01:02 (1779127262) [ 2182.287842] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2182.297543] alg: No test for adler32 (adler32-zlib) [ 2183.330896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2183.394700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2183.445299] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 2184.222822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.30@tcp [ 2184.693479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg230-client@tcp [ 2192.933058] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 14:01:13 (1779127273) [ 2193.086338] LNet: 109405:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2193.110270] LNetError: 109405:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2193.143822] LNet: Removed LNI 192.168.202.30@tcp [ 2194.050871] Key type .llcrypt unregistered [ 2194.058873] Key type ._llcrypt unregistered [ 2201.314674] Key type ._llcrypt registered [ 2201.317301] Key type .llcrypt registered [ 2201.529492] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2201.542933] alg: No test for adler32 (adler32-zlib) [ 2202.501561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2202.520931] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 2202.541326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2208.479554] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 14:01:29 (1779127289) [ 2210.236579] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2212.137436] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 14:01:33 (1779127293) [ 2213.902340] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2215.666674] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 14:01:36 (1779127296) [ 2215.818051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2215.850606] LNet: 110830:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2215.874123] LNetError: 110830:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2215.902621] LNet: Removed LNI 192.168.202.30@tcp [ 2215.922640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2216.023232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2216.048937] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 2216.755293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2216.861607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2225.394594] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 14:01:46 (1779127306) [ 2225.703936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2225.748184] LNet: 111544:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2225.757163] LNetError: 111544:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2225.776632] LNet: Removed LNI 192.168.202.30@tcp [ 2225.823642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2225.893963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2225.936606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2226.693389] Key type .llcrypt unregistered [ 2226.699168] Key type ._llcrypt unregistered [ 2239.940782] Key type ._llcrypt registered [ 2239.943618] Key type .llcrypt registered [ 2241.171598] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 14:02:02 (1779127322) [ 2241.678443] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2241.686956] alg: No test for adler32 (adler32-zlib) [ 2242.691381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2242.782271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2242.841732] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 2243.615672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2243.668785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.30@tcp [ 2250.838276] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 14:02:11 (1779127331) [ 2250.945318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2250.975744] LNet: 113097:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2250.987302] LNetError: 113097:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2251.002319] LNet: Removed LNI 192.168.202.30@tcp [ 2251.056533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2251.179793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2251.209589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2258.061848] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 14:02:19 (1779127339) [ 2258.513517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2258.578994] LNet: 113684:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2258.620706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2258.736802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2259.126702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2259.166503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2259.270558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2259.649493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2259.710670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2259.890687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2259.928645] LNet: 113717:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2259.994841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2260.157055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2260.773410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2260.834165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2260.931375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2261.577629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2261.634921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2268.425065] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 14:02:29 (1779127349) [ 2268.522561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2268.563421] LNet: 114313:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2268.598838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2268.670284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2268.700534] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 2269.428299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2269.463144] LNet: Added LNI 192.168.202.30@tcp2 [8/256/0/180] [ 2270.301993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2270.351778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.30@tcp [ 2270.531274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2270.548115] LNetError: 114597:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2270.554416] LNet: Removed LNI 192.168.202.30@tcp [ 2270.565303] LNet: Removed LNI 192.168.202.30@tcp2 [ 2270.592515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2270.664526] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 2270.673205] LNet: Added LNI 192.168.202.30@tcp2 [8/256/0/180] [ 2270.679830] LNet: 114603:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2271.288467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.30@tcp [ 2271.522420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2271.549119] LNetError: 114640:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2271.563706] LNetError: 114640:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2271.573695] LNet: Removed LNI 192.168.202.30@tcp [ 2271.602752] LNet: Removed LNI 192.168.202.30@tcp2 [ 2271.640151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2271.676357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2271.711390] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 2272.603676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2272.621786] LNet: Added LNI 192.168.202.30@tcp2 [8/256/0/180] [ 2273.500586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.30@tcp [ 2273.531553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2273.614566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.30@tcp [ 2273.868681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2273.890901] LNet: 114925:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2273.898137] LNet: 114925:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2273.906796] LNetError: 114925:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2273.912635] LNetError: 114925:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2273.919645] LNet: Removed LNI 192.168.202.30@tcp [ 2274.997188] LNet: Removed LNI 192.168.202.30@tcp2 [ 2275.048930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2275.130942] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 2275.151281] LNet: Added LNI 192.168.202.30@tcp2 [8/256/0/180] [ 2275.165679] LNet: 114931:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2275.835676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.30@tcp [ 2276.115913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2276.135595] LNet: Removed LNI 192.168.202.30@tcp [ 2277.158289] LNet: Removed LNI 192.168.202.30@tcp2 [ 2277.195773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2277.272937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2277.308199] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 2278.242648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2278.289234] LNet: Added LNI 192.168.202.30@tcp2 [8/256/0/180] [ 2279.311325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2279.402643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.30@tcp [ 2279.655108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2279.681968] LNetError: 115247:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2279.687344] LNetError: 115247:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2279.692332] LNet: Removed LNI 192.168.202.30@tcp [ 2279.707324] LNet: Removed LNI 192.168.202.30@tcp2 [ 2279.739171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2279.823326] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 2279.833474] LNet: Added LNI 192.168.202.30@tcp2 [8/256/0/180] [ 2279.840262] LNet: 115253:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2280.264884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.30@tcp [ 2280.474878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2281.503199] LNet: Removed LNI 192.168.202.30@tcp [ 2281.518754] LNet: Removed LNI 192.168.202.30@tcp2 [ 2281.586319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2281.709079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2281.740728] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 2282.594829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2282.643946] LNet: Added LNI 192.168.202.30@tcp2 [8/256/0/180] [ 2283.873548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.30@tcp [ 2283.932321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2283.996958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.30@tcp [ 2284.288618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2284.308472] LNet: 115575:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2284.311798] LNet: 115575:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2285.343337] LNet: Removed LNI 192.168.202.30@tcp [ 2286.377378] LNet: Removed LNI 192.168.202.30@tcp2 [ 2286.414186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2286.548113] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 2286.559111] LNet: Added LNI 192.168.202.30@tcp2 [8/256/0/180] [ 2286.568139] LNet: 115581:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2287.153755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.30@tcp [ 2287.380326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2287.414272] LNet: Removed LNI 192.168.202.30@tcp [ 2288.504841] LNet: Removed LNI 192.168.202.30@tcp2 [ 2288.558693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2288.691938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2288.741883] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 2289.542522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2289.568380] LNet: Added LNI 192.168.202.30@tcp2 [8/256/0/180] [ 2290.528297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2290.553157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2290.596898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.30@tcp [ 2290.899848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2290.915833] LNetError: 115915:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2290.919630] LNetError: 115915:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2290.923298] LNet: Removed LNI 192.168.202.30@tcp [ 2290.932490] LNet: Removed LNI 192.168.202.30@tcp2 [ 2290.946049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2291.019881] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 2291.027028] LNet: Added LNI 192.168.202.30@tcp2 [8/256/0/180] [ 2291.033189] LNet: 115921:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2291.520153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.30@tcp [ 2291.853745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2292.895175] LNet: Removed LNI 192.168.202.30@tcp [ 2292.908152] LNet: Removed LNI 192.168.202.30@tcp2 [ 2292.969630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2293.037528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2293.053845] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 2293.839402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2293.859714] LNet: Added LNI 192.168.202.30@tcp2 [8/256/0/180] [ 2294.657549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.30@tcp [ 2294.693453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2294.718619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2294.785137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.30@tcp [ 2295.172466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2296.223164] LNet: Removed LNI 192.168.202.30@tcp [ 2297.259452] LNet: Removed LNI 192.168.202.30@tcp2 [ 2297.308507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2297.403427] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 2297.414464] LNet: Added LNI 192.168.202.30@tcp2 [8/256/0/180] [ 2298.579071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.30@tcp [ 2306.771423] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2308.386108] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 14:03:09 (1779127389) [ 2308.603067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2308.643240] LNet: 117040:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2308.650677] LNet: 117040:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2308.658028] LNetError: 117040:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2308.665684] LNetError: 117040:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2308.676717] LNet: Removed LNI 192.168.202.30@tcp [ 2309.754846] LNet: Removed LNI 192.168.202.30@tcp2 [ 2309.841684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2310.200647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2310.217761] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 2310.931676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2310.959683] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2311.776774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2311.792747] LNet: Removed LNI 192.168.202.30@tcp [ 2311.802519] LNet: Removed LNI 10.1.2.3@tcp [ 2311.816936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2311.920581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2311.961373] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 2311.967051] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2312.494485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2312.514813] LNet: Removed LNI 192.168.202.30@tcp [ 2312.528854] LNet: Removed LNI 10.1.2.3@tcp [ 2319.076610] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 14:03:20 (1779127400) [ 2319.390918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2319.437745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2319.578038] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2319.821066] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2319.840111] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2319.850914] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2322.243552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2322.273194] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 2322.984696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2323.010973] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2325.194934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.30@tcp [ 2325.242685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2325.320108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.202.30@tcp -m PUT -e local_timeout -r 1 [ 2325.370819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2325.428235] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2325.432690] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2325.742045] LNetError: 117980:0:(lib-move.c:4710:lnet_parse()) 192.168.202.30@tcp, src 192.168.202.30@tcp: Dropping PUT (error -108 looking up sender) [ 2327.519124] LNet: Removed LNI 192.168.202.30@tcp [ 2327.530457] LNet: Removed LNI 10.1.2.3@tcp [ 2328.063324] Key type .llcrypt unregistered [ 2328.066237] Key type ._llcrypt unregistered [ 2340.795096] Key type ._llcrypt registered [ 2340.797836] Key type .llcrypt registered [ 2342.390656] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 14:03:43 (1779127423) [ 2342.988803] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2343.008781] alg: No test for adler32 (adler32-zlib) [ 2343.996119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2344.448426] LNet: 119295:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2345.020371] Key type .llcrypt unregistered [ 2345.024470] Key type ._llcrypt unregistered [ 2351.217695] Key type ._llcrypt registered [ 2351.219776] Key type .llcrypt registered [ 2351.398633] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2351.410300] alg: No test for adler32 (adler32-zlib) [ 2352.378978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2352.428609] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 2360.432701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.130@tcp [ 2360.673423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2361.288949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.130@tcp [ 2361.495774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2361.519864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2361.550544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2361.581291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2361.612355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2361.631211] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.130@tcp [ 2361.644336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2361.668643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2361.709652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2361.726495] LNet: 119747:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2361.745805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2361.773503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2361.818574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2361.847768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2361.872412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2361.899927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2362.664982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2362.912838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2362.938406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2362.975917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2363.000205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2363.033484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2363.064698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2363.096225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2363.131668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2363.159428] LNet: 119865:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2363.180770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2363.218558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2363.252698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2363.286762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2363.324280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2363.365471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2363.989936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.130@tcp [ 2364.231669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2364.260741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2364.290883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2364.318384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2364.344250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2364.371782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2364.396803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2364.424938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2364.442964] LNet: 119982:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2364.462791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2364.489810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2364.515740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2364.544648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2364.589448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2364.623694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2365.379113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2365.711925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2365.753666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2365.783733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2365.816983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2365.839591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2365.868739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2365.898512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2365.938764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2365.978475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2366.028915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2366.055761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2366.090145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2366.147627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2366.181597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2366.623114] LNet: 120145:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2366.636697] LNetError: 120145:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2366.659759] LNet: Removed LNI 192.168.202.30@tcp [ 2367.254285] Key type .llcrypt unregistered [ 2367.262888] Key type ._llcrypt unregistered [ 2383.909137] Key type ._llcrypt registered [ 2383.914340] Key type .llcrypt registered [ 2384.998674] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 14:04:26 (1779127466) [ 2385.363888] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2385.387282] alg: No test for adler32 (adler32-zlib) [ 2386.362513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2386.759805] LNet: 121036:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2387.255297] Key type .llcrypt unregistered [ 2387.257619] Key type ._llcrypt unregistered [ 2393.463327] Key type ._llcrypt registered [ 2393.465471] Key type .llcrypt registered [ 2393.665490] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2393.693298] alg: No test for adler32 (adler32-zlib) [ 2394.627623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2394.653527] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 2402.794786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.130@tcp [ 2412.480794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2412.532683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2412.570381] LNet: Added LNI 192.168.202.30@tcp1 [8/256/0/180] [ 2413.599759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2414.284504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.130@tcp [ 2414.487329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2414.518372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2414.548052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2414.597503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2414.630982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2414.657322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2414.678357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2414.700734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2414.718838] LNet: 121692:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2414.736453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2414.764560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2414.812056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2414.840540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2414.869478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2414.902847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2415.616263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2415.840626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2415.864099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2415.890885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2415.932866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2415.984182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2416.046206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2416.146549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2416.184878] LNet: 2 peer NIs in recovery (showing 2): 192.168.202.130@tcp, 192.168.202.130@tcp1 [ 2416.194995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2416.230199] LNet: 121809:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2416.249919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2416.298165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2416.346841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2416.404540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2416.459303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2416.515957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2417.073240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.130@tcp [ 2417.299608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2417.323459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2417.348136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2417.373137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2417.393162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2417.415315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2417.440401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2417.466753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2417.481384] LNet: 121927:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2417.495723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2417.524809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2417.558557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2417.609707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2417.680201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2417.741044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2418.288956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2418.516335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2418.549207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2418.580579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2418.628368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2418.668333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2418.697135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2418.732774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2418.756530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2418.781403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2418.810952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2418.836942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2418.861529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2418.885517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2418.916493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2424.284913] LNet: 122124:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2424.296946] LNetError: 122124:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2424.304867] LNet: Removed LNI 192.168.202.30@tcp [ 2424.313536] LNet: Removed LNI 192.168.202.30@tcp1 [ 2424.614192] Key type .llcrypt unregistered [ 2424.618749] Key type ._llcrypt unregistered [ 2440.951234] Key type ._llcrypt registered [ 2440.952789] Key type .llcrypt registered [ 2441.994251] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 14:05:23 (1779127523) [ 2452.544278] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 14:05:33 (1779127533) [ 2460.918663] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 14:05:41 (1779127541) [ 2468.828886] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 14:05:49 (1779127549) [ 2477.074661] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 14:05:58 (1779127558) [ 2485.613477] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 14:06:06 (1779127566) [ 2493.786534] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 14:06:14 (1779127574) [ 2501.714177] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 14:06:22 (1779127582) [ 2510.700257] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 14:06:31 (1779127591) [ 2517.735084] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 14:06:38 (1779127598) [ 2525.684848] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 14:06:46 (1779127606) [ 2533.919285] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 14:06:55 (1779127615) [ 2540.963887] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 14:07:01 (1779127621) [ 2548.201918] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 14:07:09 (1779127629) [ 2555.845706] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 14:07:16 (1779127636) [ 2563.105794] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 14:07:24 (1779127644) [ 2569.952518] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 14:07:31 (1779127651) [ 2576.031764] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 14:07:37 (1779127657) [ 2576.500402] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2576.517751] alg: No test for adler32 (adler32-zlib) [ 2577.401835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2577.509734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2577.524472] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 2578.412570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2578.437644] LNet: 132839:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2578.444337] LNetError: 132839:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2578.455214] LNet: Removed LNI 192.168.202.30@tcp [ 2578.474488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2578.534753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2578.554459] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 2584.011267] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 14:07:45 (1779127665) [ 2590.703588] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 14:07:52 (1779127672) [ 2596.908379] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 14:07:58 (1779127678) [ 2603.845001] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 14:08:05 (1779127685) [ 2610.097043] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 14:08:11 (1779127691) [ 2616.915864] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 14:08:18 (1779127698) [ 2623.489307] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 14:08:24 (1779127704) [ 2629.477502] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 14:08:30 (1779127710) [ 2636.700504] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 14:08:37 (1779127717) [ 2644.576713] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 14:08:45 (1779127725) [ 2652.764919] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 14:08:53 (1779127733) [ 2660.374771] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 14:09:01 (1779127741) [ 2668.136372] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2441 sec ========== 14:09:08 (1779127748) [ 2668.609744] LNet: 140303:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2668.617795] LNetError: 140303:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2668.636726] LNet: Removed LNI 192.168.202.30@tcp [ 2669.378637] Key type .llcrypt unregistered [ 2669.383904] Key type ._llcrypt unregistered [ 2682.621435] Key type ._llcrypt registered [ 2682.640042] Key type .llcrypt registered [ 2682.934834] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2682.952781] alg: No test for adler32 (adler32-zlib) [ 2684.147785] Lustre: Lustre: Build Version: 2.17.53_24_g2ff46d7 [ 2684.396055] LNet: Added LNI 192.168.202.30@tcp [8/256/0/180] [ 2686.016397] Key type lgssc registered [ 2686.811351] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2754.763345] Lustre: Mounted lustre-client [ 2759.173463] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2767.319445] LustreError: 142788:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffff919f91029000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 2767.412960] Lustre: Unmounted lustre-client [ 2806.112582] Key type lgssc unregistered [ 2806.376513] LNet: 143271:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2806.382629] LNetError: 143271:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2806.401830] LNet: Removed LNI 192.168.202.30@tcp [ 2807.091953] Key type .llcrypt unregistered [ 2807.097606] Key type ._llcrypt unregistered