[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 442659688 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.002311] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004017] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007032] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008020] pid_max: default: 32768 minimum: 301 [ 0.010019] LSM: Security Framework initializing [ 0.011062] Yama: becoming mindful. [ 0.012045] SELinux: Initializing. [ 0.013095] *** VALIDATE selinux *** [ 0.021028] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026238] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027155] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028134] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029164] *** VALIDATE tmpfs *** [ 0.031098] *** VALIDATE proc *** [ 0.032340] *** VALIDATE cgroup *** [ 0.033019] *** VALIDATE cgroup2 *** [ 0.034344] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035196] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036017] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037048] Spectre V2 : User space: Vulnerable [ 0.038015] Speculative Store Bypass: Vulnerable [ 0.041821] debug: unmapping init [mem 0xffffffff8d059000-0xffffffff8d060fff] [ 0.043956] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044709] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045032] ... version: 2 [ 0.046018] ... bit width: 48 [ 0.047013] ... generic registers: 4 [ 0.048016] ... value mask: 0000ffffffffffff [ 0.049017] ... max period: 00007fffffffffff [ 0.050019] ... fixed-purpose events: 3 [ 0.051018] ... event mask: 000000070000000f [ 0.052388] rcu: Hierarchical SRCU implementation. [ 0.054568] smp: Bringing up secondary CPUs ... [ 0.055622] x86: Booting SMP configuration: [ 0.056044] .... node #0, CPUs: #1 #2 #3 [ 0.059706] smp: Brought up 1 node, 4 CPUs [ 0.061019] smpboot: Max logical packages: 1 [ 0.062027] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.230103] node 0 deferred pages initialised in 165ms [ 0.232610] devtmpfs: initialized [ 0.233273] x86/mm: Memory block size: 128MB [ 0.235813] gcov: version magic: 0x41383552 [ 0.237477] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.240091] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.243333] pinctrl core: initialized pinctrl subsystem [ 0.244132] [ 0.244593] ************************************************************* [ 0.248013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.250012] ** ** [ 0.252011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.255014] ** ** [ 0.257013] ** This means that this kernel is built to expose internal ** [ 0.259013] ** IOMMU data structures, which may compromise security on ** [ 0.261014] ** your system. ** [ 0.264014] ** ** [ 0.266013] ** If you see this message and you are not debugging the ** [ 0.268015] ** kernel, report this immediately to your vendor! ** [ 0.270015] ** ** [ 0.273015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.275012] ************************************************************* [ 0.278715] NET: Registered protocol family 16 [ 0.280456] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.283069] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.286075] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.289495] cpuidle: using governor menu [ 0.290616] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.292671] PCI: Using configuration type 1 for base access [ 0.295244] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.305118] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.307038] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.310139] cryptd: max_cpu_qlen set to 1000 [ 0.314151] ACPI: Added _OSI(Module Device) [ 0.315015] ACPI: Added _OSI(Processor Device) [ 0.317016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.318017] ACPI: Added _OSI(Processor Aggregator Device) [ 0.323186] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.329634] ACPI: Interpreter enabled [ 0.330063] ACPI: PM: (supports S0 S3 S4 S5) [ 0.332016] ACPI: Using IOAPIC for interrupt routing [ 0.334228] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.338461] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.349065] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.351065] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.354030] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.356111] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.361364] acpiphp: Slot [2] registered [ 0.363119] acpiphp: Slot [5] registered [ 0.364104] acpiphp: Slot [6] registered [ 0.365090] acpiphp: Slot [3] registered [ 0.368139] acpiphp: Slot [4] registered [ 0.369236] acpiphp: Slot [7] registered [ 0.371145] acpiphp: Slot [8] registered [ 0.373177] acpiphp: Slot [9] registered [ 0.374096] acpiphp: Slot [10] registered [ 0.375207] acpiphp: Slot [11] registered [ 0.377115] acpiphp: Slot [12] registered [ 0.378130] acpiphp: Slot [13] registered [ 0.380144] acpiphp: Slot [14] registered [ 0.381162] acpiphp: Slot [15] registered [ 0.383216] acpiphp: Slot [16] registered [ 0.384130] acpiphp: Slot [17] registered [ 0.386121] acpiphp: Slot [18] registered [ 0.387171] acpiphp: Slot [19] registered [ 0.389160] acpiphp: Slot [20] registered [ 0.391134] acpiphp: Slot [21] registered [ 0.392198] acpiphp: Slot [22] registered [ 0.394089] acpiphp: Slot [23] registered [ 0.395152] acpiphp: Slot [24] registered [ 0.397138] acpiphp: Slot [25] registered [ 0.398108] acpiphp: Slot [26] registered [ 0.399154] acpiphp: Slot [27] registered [ 0.401122] acpiphp: Slot [28] registered [ 0.402156] acpiphp: Slot [29] registered [ 0.404133] acpiphp: Slot [30] registered [ 0.406162] acpiphp: Slot [31] registered [ 0.407082] PCI host bridge to bus 0000:00 [ 0.409032] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.412033] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.414031] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.417034] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.420036] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.423034] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.425296] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.429650] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.433479] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.438986] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.443045] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.447026] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.450035] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.453024] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.456618] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.459119] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.462056] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.465979] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.470018] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.480851] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.484019] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.489887] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.498104] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.504017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.516018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.525288] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.532023] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.536983] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.566020] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.575597] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.577355] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.579373] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.582402] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.584231] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.589049] iommu: Default domain type: Passthrough [ 0.591458] SCSI subsystem initialized [ 0.592143] ACPI: bus type USB registered [ 0.593105] usbcore: registered new interface driver usbfs [ 0.595069] usbcore: registered new interface driver hub [ 0.596080] usbcore: registered new device driver usb [ 0.598211] pps_core: LinuxPPS API ver. 1 registered [ 0.599010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.601064] PTP clock support registered [ 0.603115] EDAC MC: Ver: 3.0.0 [ 0.605126] PCI: Using ACPI for IRQ routing [ 0.606785] NetLabel: Initializing [ 0.608011] NetLabel: domain hash size = 128 [ 0.609008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.610132] NetLabel: unlabeled traffic allowed by default [ 0.612175] vgaarb: loaded [ 0.613257] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.615019] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.619650] clocksource: Switched to clocksource kvm-clock [ 0.726165] VFS: Disk quotas dquot_6.6.0 [ 0.727576] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.729766] *** VALIDATE ramfs *** [ 0.730971] *** VALIDATE hugetlbfs *** [ 0.733443] pnp: PnP ACPI init [ 0.736237] pnp: PnP ACPI: found 6 devices [ 0.754055] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.757285] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.759309] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.761312] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.763529] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.765689] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.768185] NET: Registered protocol family 2 [ 0.770664] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.775205] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.778555] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.783954] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.787671] TCP: Hash tables configured (established 65536 bind 65536) [ 0.790584] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.793167] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.795848] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.798780] NET: Registered protocol family 1 [ 0.801521] RPC: Registered named UNIX socket transport module. [ 0.803639] RPC: Registered udp transport module. [ 0.805309] RPC: Registered tcp transport module. [ 0.806762] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.808826] NET: Registered protocol family 44 [ 0.810674] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.812564] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.814548] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.816627] PCI: CLS 0 bytes, default 64 [ 0.818083] Unpacking initramfs... [ 2.227748] debug: unmapping init [mem 0xffff8e61bcc64000-0xffff8e61bffcffff] [ 2.231827] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.233939] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.236671] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.747919] Initialise system trusted keyrings [ 2.749841] Key type blacklist registered [ 2.751714] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.778699] zbud: loaded [ 2.782643] *** VALIDATE nfs *** [ 2.784184] *** VALIDATE nfs4 *** [ 2.785938] pstore: using deflate compression [ 2.789718] Platform Keyring initialized [ 2.896766] NET: Registered protocol family 38 [ 2.898703] Key type asymmetric registered [ 2.900228] Asymmetric key parser 'x509' registered [ 2.901942] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.905148] io scheduler mq-deadline registered [ 2.907572] io scheduler kyber registered [ 2.909327] io scheduler bfq registered [ 2.911689] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.914518] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.918153] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.921975] ACPI: Power Button [PWRF] [ 2.927605] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.934144] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.946184] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.982447] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.019692] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.038306] Non-volatile memory driver v1.3 [ 3.040285] Linux agpgart interface v0.103 [ 3.076780] virtio_blk virtio1: [vda] 136536 512-byte logical blocks (69.9 MB/66.7 MiB) [ 3.080120] vda: detected capacity change from 0 to 69906432 [ 3.097396] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.100058] vdb: detected capacity change from 0 to 1073741824 [ 3.109173] libphy: Fixed MDIO Bus: probed [ 3.124713] usbcore: registered new interface driver usbserial_generic [ 3.128600] usbserial: USB Serial support registered for generic [ 3.130675] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.135266] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.136696] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.139735] mousedev: PS/2 mouse device common for all mice [ 3.142910] rtc_cmos 00:05: RTC can wake from S4 [ 3.145091] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.147220] rtc_cmos 00:05: registered as rtc0 [ 3.151285] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.152057] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.155950] intel_pstate: CPU model not supported [ 3.159203] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.164319] hid: raw HID events driver (C) Jiri Kosina [ 3.166463] usbcore: registered new interface driver usbhid [ 3.168078] usbhid: USB HID core driver [ 3.169427] drop_monitor: Initializing network drop monitor service [ 3.171552] Initializing XFRM netlink socket [ 3.173531] NET: Registered protocol family 10 [ 3.176333] Segment Routing with IPv6 [ 3.178198] NET: Registered protocol family 17 [ 3.180314] mpls_gso: MPLS GSO support [ 3.188250] RAS: Correctable Errors collector initialized. [ 3.190548] AVX version of gcm_enc/dec engaged. [ 3.192306] AES CTR mode by8 optimization enabled [ 3.285847] sched_clock: Marking stable (3285753406, 0)->(4278226062, -992472656) [ 3.289832] registered taskstats version 1 [ 3.292909] Loading compiled-in X.509 certificates [ 3.295282] zswap: loaded using pool lzo/zbud [ 3.325096] Key type big_key registered [ 3.340529] Key type encrypted registered [ 3.342489] ima: No TPM chip found, activating TPM-bypass! [ 3.345408] ima: Allocated hash algorithm: sha1 [ 3.347547] ima: No architecture policies found [ 3.349512] evm: Initialising EVM extended attributes: [ 3.351384] evm: security.selinux [ 3.352406] evm: security.ima [ 3.353236] evm: security.capability [ 3.354192] evm: HMAC attrs: 0x1 [ 3.356286] rtc_cmos 00:05: setting system clock to 2026-06-12 00:57:41 UTC (1781225861) [ 3.363228] debug: unmapping init [mem 0xffffffff8e003000-0xffffffff8e1fffff] [ 3.366242] debug: unmapping init [mem 0xffffffff8cd82000-0xffffffff8d058fff] [ 3.375143] Write protecting the kernel read-only data: 28672k [ 3.378391] debug: unmapping init [mem 0xffffffff8b403000-0xffffffff8b5fffff] [ 3.381387] debug: unmapping init [mem 0xffffffff8bd14000-0xffffffff8bdfffff] [ 3.417944] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.426941] systemd[1]: Detected virtualization kvm. [ 3.428741] systemd[1]: Detected architecture x86-64. [ 3.430729] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.456971] systemd[1]: No hostname configured. [ 3.458549] systemd[1]: Set hostname to . [ 3.460310] random: systemd: uninitialized urandom read (16 bytes read) [ 3.462405] systemd[1]: Initializing machine ID from random generator. [ 3.513494] random: ln: uninitialized urandom read (6 bytes read) [ 3.610296] random: systemd: uninitialized urandom read (16 bytes read) [ 3.612679] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.618610] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.630666] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Paths. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.335185] device-mapper: uevent: version 1.0.3 [ 4.337482] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.090511] virtio_net virtio0 ens2: renamed from eth0 [ 5.419255] scsi host0: ata_piix [ 5.486361] scsi host1: ata_piix [ 5.491363] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.496146] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.882984] random: crng init done [ 9.886585] random: 7 urandom warning(s) missed due to ratelimiting [ 11.253646] dracut-initqueue[578]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 13.946591] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 17.033663] printk: systemd: 26 output lines suppressed due to ratelimiting [ 17.771754] SELinux: Disabled at runtime. [ 17.926925] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 17.953143] systemd[1]: Detected virtualization kvm. [ 17.955816] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 19.359278] systemd[1]: initrd-switch-root.service: Succeeded. [ 19.364924] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 19.384581] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 19.395512] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 19.399855] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 19.413811] systemd[1]: Starting Journal Service... Starting Journal Service... [ 19.432967] systemd[1]: Listening on RPCbind Server Activation Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target rpc_pipefs.target. Activating swap /dev/disk/by-label/SWAP... Starting Create list of required st…ce nodes for the current kernel... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. [ 19.512685] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Mounting Kernel Debug File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Mounting POSIX Message Queue File System... Starting Apply Kernel Variables... Starting Remount Root and Kernel File Systems... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting Huge Pages File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 20.857320] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 21.947810] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 22.007940] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 22.840439] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 22.939349] EDAC sbridge: Ver: 1.1.2 [ 26.589069] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (7s / no limit)[ 27.229954] NFS: Registering the id_resolver key type [ 27.236533] Key type id_resolver registered [ 27.242480] Key type id_legacy registered [*** ] A start job is running for Configur…-only root support (8s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Login Service. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg322-client login: [ 55.627412] hrtimer: interrupt took 2732706 ns [ 96.664132] libcfs: loading out-of-tree module taints kernel. [ 96.831120] Key type ._llcrypt registered [ 96.868846] Key type .llcrypt registered [ 97.400192] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 97.420465] alg: No test for adler32 (adler32-zlib) [ 99.061621] Lustre: Lustre: Build Version: 2.17.53_68_ga4ebd63 [ 100.164543] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 102.008430] Key type lgssc registered [ 104.061889] Lustre: Echo OBD driver; http://www.lustre.org/ [ 225.408585] Lustre: Mounted lustre-client [ 231.139579] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 244.573534] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing check_logdir /tmp/testlogs/ [ 248.885563] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing yml_node [ 250.848332] Lustre: lustre-OST0000-osc-ffff8e6210493000: disconnect after 23s idle [ 254.215238] Lustre: DEBUG MARKER: Client: 2.17.53.68 [ 256.577602] Lustre: DEBUG MARKER: MDS: 2.17.53.68 [ 259.529820] Lustre: DEBUG MARKER: OSS: 2.17.53.68 [ 261.506919] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Thu Jun 11 21:01:57 EDT 2026 [ 279.567155] Lustre: Unmounted lustre-client [ 317.279606] Key type lgssc unregistered [ 317.619040] LNet: 8395:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 317.626505] LNetError: 8395:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 317.639466] LNet: Removed LNI 192.168.203.22@tcp [ 318.748188] Key type .llcrypt unregistered [ 318.756143] Key type ._llcrypt unregistered [ 347.374064] Key type ._llcrypt registered [ 347.378159] Key type .llcrypt registered [ 347.557484] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 347.574948] alg: No test for adler32 (adler32-zlib) [ 348.449981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 348.527278] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 348.550752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 352.332946] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 352.514085] LNet: 9279:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 352.539617] LNetError: 9279:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 352.575661] LNet: Removed LNI 192.168.203.22@tcp [ 353.949579] Key type .llcrypt unregistered [ 353.957417] Key type ._llcrypt unregistered [ 360.570482] Key type ._llcrypt registered [ 360.575289] Key type .llcrypt registered [ 362.142864] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 21:03:38 (1781226218) [ 362.496225] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 362.507723] alg: No test for adler32 (adler32-zlib) [ 363.404446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 363.448614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 363.464191] LNet: 9693:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 363.475706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 363.512460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 363.528660] LNet: 9700:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 368.902901] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 21:03:45 (1781226225) [ 369.004080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 369.029324] LNet: 10275:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 369.060179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 369.118530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 374.919843] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 21:03:51 (1781226231) [ 375.014254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 375.052491] LNet: 10863:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 375.081277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 375.182866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 381.755633] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 21:03:58 (1781226238) [ 382.025891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 382.041675] LNet: 11447:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 382.090555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 382.307607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 392.421972] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 21:04:08 (1781226248) [ 392.547838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 392.616649] LNet: 12026:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 392.668717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 392.877253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 393.474320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 401.515742] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 21:04:18 (1781226258) [ 401.785698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 401.804833] LNet: 12624:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 401.840536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 401.904444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 402.155440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 402.588374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 408.519937] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 21:04:25 (1781226265) [ 408.613457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 408.666534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 408.781625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 414.636958] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 21:04:31 (1781226271) [ 414.840927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 414.882347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 414.932459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 414.963876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 415.500238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 415.588196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 415.974338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 416.023701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 416.467962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 416.540725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 416.953909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 417.021242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 417.398732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 417.440774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 417.919290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 417.991213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 418.739940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 418.792381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 419.191121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 419.237435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 419.806713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 419.870581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 420.675983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 420.737610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 421.504962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 421.643171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 422.146392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 422.195341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 430.215815] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 21:04:46 (1781226286) [ 430.362876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 430.413348] LNet: 14485:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 430.424923] LNet: 14485:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 430.484096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 430.573882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 430.633220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 437.885649] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 21:04:54 (1781226294) [ 437.999969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 438.041409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 438.129614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 438.185596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 444.649621] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 21:05:01 (1781226301) [ 444.842589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 444.893366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 445.027183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 445.087308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 451.459483] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 21:05:08 (1781226308) [ 451.771955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 451.820942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 451.958973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 452.002972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 458.869473] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 21:05:15 (1781226315) [ 459.006708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 459.079692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 459.139597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 459.187630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 465.314430] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 21:05:22 (1781226322) [ 465.494631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 465.527392] LNet: 17414:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 465.534709] LNet: 17414:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 465.566160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 465.682898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 465.704658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 471.468117] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 21:05:28 (1781226328) [ 471.608525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 471.664880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 471.784617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 472.209276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 478.605955] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 21:05:35 (1781226335) [ 478.710354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 478.739955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 479.014235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 479.244447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 485.525928] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 21:05:42 (1781226342) [ 485.820033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 485.861334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 485.994495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 486.328370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 492.769818] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 21:05:49 (1781226349) [ 493.047293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 493.101438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 493.697441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 494.012855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 500.947979] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 21:05:57 (1781226357) [ 501.029356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 501.092765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 501.202220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 501.270570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 507.467879] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 21:06:04 (1781226364) [ 507.550138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 507.611822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 507.687061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 507.726472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 514.137346] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 21:06:10 (1781226370) [ 514.196323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 514.237686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 514.330085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 519.861345] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 21:06:16 (1781226376) [ 519.997826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 520.051538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 520.117651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 525.843683] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 21:06:22 (1781226382) [ 526.041507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 526.093817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 526.199614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 526.632139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 532.743330] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 21:06:29 (1781226389) [ 532.854804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 532.869448] LNet: 23441:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 532.876247] LNet: 23441:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 532.904144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 533.059933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 533.087409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 539.437719] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 21:06:36 (1781226396) [ 539.551110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 539.623882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 539.690980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 539.715419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 546.831608] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 21:06:43 (1781226403) [ 547.037723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 547.090296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 547.178116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 547.223843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 553.750326] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 21:06:50 (1781226410) [ 553.984362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 554.039875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 554.144971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 554.216321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 561.211616] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 21:06:58 (1781226418) [ 561.417905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 561.462229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 561.533460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 561.608873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 561.630678] LNetError: 25801:0:(peer.c:2175:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 561.697696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 561.741644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 567.738814] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 21:07:04 (1781226424) [ 567.909879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 568.000793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 568.081279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.ul0RVtqAw6 [ 568.169644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.ul0RVtqAw6 [ 568.214807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.ul0RVtqAw6 [ 574.747102] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 21:07:11 (1781226431) [ 574.862947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 574.931085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 575.017734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 575.053697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 575.100183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 581.116843] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 21:07:17 (1781226437) [ 581.290859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 581.350486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 581.439710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 584.462212] LNet: Message forwarding will be enabled in 2s [ 587.552188] LNet: Message forwarding enabled [ 588.002177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 588.022205] LNet: Message forwarding will stop in 2s to 6s [ 592.584070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 593.245134] LNet: Message forwarding disabled [ 594.951308] LNet: Message forwarding will be enabled in 2s [ 598.368138] LNet: Message forwarding enabled [ 602.087271] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 21:07:38 (1781226458) [ 602.309349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 603.695043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 603.778421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 610.592150] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 21:07:47 (1781226467) [ 610.710182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 610.782156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 610.894170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 610.927177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 610.962311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 610.993815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 611.031232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 611.074876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 611.127358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 611.154818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 611.184770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 611.233839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 611.278711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 611.334800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 611.397923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 611.437786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 611.478873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 611.526932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 611.572026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 611.624870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 611.655309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 611.696897] LNetError: 28870:0:(router.c:624:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 618.428088] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 21:07:55 (1781226475) [ 618.622481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 618.694980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 618.771233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 625.741798] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 21:08:02 (1781226482) [ 627.598885] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg322-server [ 629.432614] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 21:08:06 (1781226486) [ 629.547757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 629.603494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 629.717599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 629.749697] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 631.569772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.203.[23-27]@tcp [ 631.618681] LNet: 30586:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 631.669631] LNetError: 30374:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.23@tcp is being used as a gateway but routing feature is not turned on [ 631.689151] LNetError: 30374:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.23@tcp has gone from up to down [ 632.679265] LNetError: 30374:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.26@tcp is being used as a gateway but routing feature is not turned on [ 632.699972] LNetError: 30374:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 633.714980] LNetError: 30374:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.27@tcp is being used as a gateway but routing feature is not turned on [ 633.730559] LNetError: 30374:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 635.747956] LNetError: 30374:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.23@tcp is being used as a gateway but routing feature is not turned on [ 635.769031] LNetError: 30374:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 638.370585] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 21:08:15 (1781226495) [ 639.853065] LNetError: 30374:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.24@tcp is being used as a gateway but routing feature is not turned on [ 639.870429] LNetError: 30374:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 640.663237] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg322-server [ 642.441707] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 21:08:19 (1781226499) [ 642.644952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 642.691718] LNetError: 31521:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 642.720908] LNet: Removed LNI 192.168.203.22@tcp [ 642.800044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 642.887687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 642.909728] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 643.961888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.203.[23-28/2]@tcp [ 643.997503] LNet: 31638:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 644.008086] LNet: 31638:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 644.039809] LNetError: 31525:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp103 through 192.168.203.25@tcp has gone from up to down [ 644.053928] LNetError: 31525:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 644.057306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.203.[23-28/2]@tcp [ 651.185937] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 21:08:27 (1781226507) [ 651.371970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 651.401295] LNetError: 32229:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 651.435855] LNet: Removed LNI 192.168.203.22@tcp [ 651.474357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 651.560548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 651.648570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 651.690543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 651.766666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 651.839564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 651.877274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 651.974978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 652.067833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 652.108867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 652.190930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 652.271749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 652.311925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 652.389111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 652.473246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 652.531955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 652.618852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 652.695163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 652.738423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 652.784793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 652.884722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 652.947367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 653.054554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 653.211227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 653.237929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 653.310068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 653.392738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 653.451674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 653.538925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 653.654989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 653.711820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 653.757535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 653.822383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 653.866331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 653.928505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 659.782659] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 21:08:36 (1781226516) [ 661.437727] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg322-server [ 662.880536] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 21:08:39 (1781226519) [ 664.960477] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg322-server [ 666.521622] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 21:08:43 (1781226523) [ 666.617777] LNet: 33623:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 666.629715] LNet: 33623:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 24 previous similar messages [ 667.257729] Key type .llcrypt unregistered [ 667.260460] Key type ._llcrypt unregistered [ 673.192633] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 673.333273] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 673.335337] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 673.354769] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 676.046745] Key type ._llcrypt registered [ 676.052304] Key type .llcrypt registered [ 676.296723] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 676.321266] alg: No test for adler32 (adler32-zlib) [ 677.297932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 677.388822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 677.438988] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 678.008824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 678.027463] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 679.944450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 679.962164] LNet: Removed LNI 10.1.2.3@tcp [ 680.082644] LNet: 34353:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 680.098634] LNetError: 34353:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 680.138856] LNet: Removed LNI 192.168.203.22@tcp [ 680.936588] Key type .llcrypt unregistered [ 680.944264] Key type ._llcrypt unregistered [ 692.019945] Key type ._llcrypt registered [ 692.026280] Key type .llcrypt registered [ 693.569590] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 21:09:09 (1781226549) [ 694.307270] Key type .llcrypt unregistered [ 694.311775] Key type ._llcrypt unregistered [ 700.867227] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 701.031178] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 701.035185] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 701.037904] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 703.491155] Key type ._llcrypt registered [ 703.499535] Key type .llcrypt registered [ 703.692986] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 703.708716] alg: No test for adler32 (adler32-zlib) [ 704.644904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 704.695786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 704.716371] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 705.346556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 705.366494] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 707.517892] LNet: 35846:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 707.522655] LNetError: 35846:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 707.526942] LNet: Removed LNI 192.168.203.22@tcp [ 707.539473] LNet: Removed LNI 10.1.2.3@tcp [ 708.146155] Key type .llcrypt unregistered [ 708.150556] Key type ._llcrypt unregistered [ 719.011398] Key type ._llcrypt registered [ 719.012987] Key type .llcrypt registered [ 720.069940] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 21:09:37 (1781226577) [ 720.606325] Key type .llcrypt unregistered [ 720.612174] Key type ._llcrypt unregistered [ 726.025569] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 726.096875] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 726.105138] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 726.115923] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 728.890258] Key type ._llcrypt registered [ 728.896804] Key type .llcrypt registered [ 729.087231] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 729.114384] alg: No test for adler32 (adler32-zlib) [ 730.027791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 730.086502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 730.107964] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 732.174697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 732.215394] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 734.070314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 734.093325] LNet: Removed LNI 10.1.2.3@tcp [ 734.112277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 734.149025] LNet: Removed LNI 10.1.2.31@tcp [ 734.167580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 734.210404] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 735.003286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 735.035653] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 735.899302] LNet: 37774:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 735.904387] LNetError: 37774:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 735.908115] LNet: Removed LNI 10.1.2.3@tcp [ 735.935639] LNet: Removed LNI 10.1.2.31@tcp [ 736.536538] Key type .llcrypt unregistered [ 736.541201] Key type ._llcrypt unregistered [ 746.568718] Key type ._llcrypt registered [ 746.571395] Key type .llcrypt registered [ 747.801876] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 21:10:04 (1781226604) [ 748.307891] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 748.341682] alg: No test for adler32 (adler32-zlib) [ 749.241116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 749.293249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 749.313835] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 750.268815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 750.301430] LNet: 38731:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 750.310826] LNetError: 38731:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 750.335099] LNet: Removed LNI 192.168.203.22@tcp [ 750.361546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 750.417654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.203.22@tcp [ 750.439108] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 751.135310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.203.22@tcp [ 751.174843] LNet: Removed LNI 192.168.203.22@tcp [ 751.233465] LNet: 38867:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 751.835499] Key type .llcrypt unregistered [ 751.838148] Key type ._llcrypt unregistered [ 761.811291] Key type ._llcrypt registered [ 761.817537] Key type .llcrypt registered [ 763.097155] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 21:10:19 (1781226619) [ 764.606490] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg322-server [ 766.127451] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 21:10:23 (1781226623) [ 766.772673] Key type .llcrypt unregistered [ 766.775039] Key type ._llcrypt unregistered [ 772.130477] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 772.240461] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 772.252962] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 772.261941] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 774.842807] Key type ._llcrypt registered [ 774.845472] Key type .llcrypt registered [ 774.981282] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 774.992785] alg: No test for adler32 (adler32-zlib) [ 775.886294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 775.955918] LNet: Added LNI 192.168.203.22@tcp1 [8/256/0/180] [ 776.753511] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 778.896336] LNet: 40697:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 778.910538] LNetError: 40697:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 778.919934] LNet: Removed LNI 192.168.203.22@tcp1 [ 778.943063] LNet: Removed LNI 10.1.2.3@tcp2 [ 779.615577] Key type .llcrypt unregistered [ 779.618271] Key type ._llcrypt unregistered [ 789.332443] Key type ._llcrypt registered [ 789.333991] Key type .llcrypt registered [ 790.258895] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 21:10:47 (1781226647) [ 790.586042] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 790.604864] alg: No test for adler32 (adler32-zlib) [ 791.504964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 791.545494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 791.559458] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 792.188248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 792.228703] LNet: 41666:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 792.243556] LNetError: 41666:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 792.262833] LNet: Removed LNI 192.168.203.22@tcp [ 792.289804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 792.358364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 792.389555] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 799.411651] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 21:10:55 (1781226655) [ 799.642703] LNet: 42256:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 799.656447] LNetError: 42256:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 799.697646] LNet: Removed LNI 192.168.203.22@tcp [ 800.374537] Key type .llcrypt unregistered [ 800.380507] Key type ._llcrypt unregistered [ 806.512078] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 806.689384] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 806.702736] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 806.722715] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 809.287485] Key type ._llcrypt registered [ 809.294400] Key type .llcrypt registered [ 809.569742] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 809.600650] alg: No test for adler32 (adler32-zlib) [ 810.499483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 810.551262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 810.574760] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 811.558943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 811.616813] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 814.516874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 814.538146] LNet: 42999:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 814.542974] LNetError: 42999:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 814.548666] LNet: Removed LNI 192.168.203.22@tcp [ 814.563641] LNet: Removed LNI 10.1.2.3@tcp [ 814.586565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 814.671450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 814.691330] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 814.696772] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 815.192042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 815.239375] LNet: 43026:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 815.244236] LNetError: 43026:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 815.248455] LNetError: 43026:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 815.254184] LNet: Removed LNI 192.168.203.22@tcp [ 815.275232] LNet: Removed LNI 10.1.2.3@tcp [ 815.317508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 815.425781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 815.465458] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 815.474413] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 816.129597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 816.156088] LNet: Added LNI 192.168.203.22@tcp2 [8/256/0/180] [ 817.508978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 817.570928] LNet: 43188:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 817.576552] LNetError: 43188:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 817.589417] LNetError: 43188:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 817.604963] LNet: Removed LNI 192.168.203.22@tcp [ 817.611621] LNet: Removed LNI 10.1.2.3@tcp [ 817.635736] LNet: Removed LNI 192.168.203.22@tcp2 [ 817.721585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 817.792715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 817.837606] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 817.840619] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 817.850477] LNet: Added LNI 192.168.203.22@tcp2 [8/256/0/180] [ 825.473513] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 21:11:22 (1781226682) [ 825.800164] LNet: 43787:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 825.806824] LNetError: 43787:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 825.809514] LNetError: 43787:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 825.817291] LNet: Removed LNI 192.168.203.22@tcp [ 825.820141] LNet: Removed LNI 10.1.2.3@tcp [ 825.827330] LNet: Removed LNI 192.168.203.22@tcp2 [ 826.522697] Key type .llcrypt unregistered [ 826.524541] Key type ._llcrypt unregistered [ 832.269273] Key type ._llcrypt registered [ 832.271893] Key type .llcrypt registered [ 832.427837] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 832.444302] alg: No test for adler32 (adler32-zlib) [ 833.344829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 833.399987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 833.421424] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 834.189869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 834.205498] LNet: 44182:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 834.209990] LNetError: 44182:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 834.220947] LNet: Removed LNI 192.168.203.22@tcp [ 834.241306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 834.301356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 834.316979] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 834.598363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 834.618537] LNet: Removed LNI 192.168.203.22@tcp [ 834.644172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 834.710591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 834.747179] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 835.295820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 835.336937] LNet: 44242:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 835.342585] LNet: 44242:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 835.353533] LNetError: 44242:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 835.362278] LNetError: 44242:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 835.391731] LNet: Removed LNI 192.168.203.22@tcp [ 835.411739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 835.503165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 835.521981] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 836.026993] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 836.154373] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 836.161933] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 836.169820] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 838.626514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 838.653679] LNet: 44321:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 838.659711] LNetError: 44321:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 838.678677] LNet: Removed LNI 192.168.203.22@tcp [ 838.711696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 838.823185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 838.865679] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 845.857899] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 21:11:42 (1781226702) [ 846.057584] LNet: 44918:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 846.065620] LNetError: 44918:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 846.083509] LNet: Removed LNI 192.168.203.22@tcp [ 846.699968] Key type .llcrypt unregistered [ 846.705932] Key type ._llcrypt unregistered [ 853.268847] Key type ._llcrypt registered [ 853.271669] Key type .llcrypt registered [ 853.526404] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 853.552226] alg: No test for adler32 (adler32-zlib) [ 854.499264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 854.571946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 854.604550] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 855.535156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 855.565567] LNet: 45316:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 855.572308] LNetError: 45316:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 855.599875] LNet: Removed LNI 192.168.203.22@tcp [ 855.638256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 855.735788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 855.760256] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 862.549956] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 21:11:59 (1781226719) [ 862.789685] LNet: 45908:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 862.800446] LNetError: 45908:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 862.841287] LNet: Removed LNI 192.168.203.22@tcp [ 863.438496] Key type .llcrypt unregistered [ 863.441417] Key type ._llcrypt unregistered [ 868.275926] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 868.493282] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 868.497300] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 868.503315] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 870.992871] Key type ._llcrypt registered [ 870.994465] Key type .llcrypt registered [ 871.215484] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 871.241233] alg: No test for adler32 (adler32-zlib) [ 872.185942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 872.257775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 872.286387] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 873.312247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 873.328850] LNet: 46357:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 873.339919] LNetError: 46357:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 873.351710] LNet: Removed LNI 192.168.203.22@tcp [ 873.391410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 873.453980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 873.485894] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 880.014941] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 21:12:16 (1781226736) [ 880.165870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 880.215786] LNet: 46954:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 880.228945] LNetError: 46954:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 880.249332] LNet: Removed LNI 192.168.203.22@tcp [ 880.276091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 880.371318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 885.778473] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 21:12:22 (1781226742) [ 885.892140] LNet: 47530:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 886.487531] Key type .llcrypt unregistered [ 886.489913] Key type ._llcrypt unregistered [ 892.655497] Key type ._llcrypt registered [ 892.664621] Key type .llcrypt registered [ 892.895207] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 892.941561] alg: No test for adler32 (adler32-zlib) [ 893.880098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 893.939247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 893.957212] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 894.736165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 894.749897] LNet: 47935:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 894.755913] LNetError: 47935:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 894.767654] LNet: Removed LNI 192.168.203.22@tcp [ 894.791246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 894.842310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 894.866098] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 901.219784] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 21:12:37 (1781226757) [ 901.297097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 901.320434] LNet: 48521:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 901.326587] LNetError: 48521:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 901.352119] LNet: Removed LNI 192.168.203.22@tcp [ 901.374538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 901.481324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 906.981976] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 21:12:43 (1781226763) [ 908.275085] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 909.703695] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 21:12:46 (1781226766) [ 910.904160] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 912.179284] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 21:12:49 (1781226769) [ 913.560176] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 914.810130] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 21:12:51 (1781226771) [ 916.040587] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 917.837450] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 21:12:54 (1781226774) [ 919.497272] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 921.339180] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 21:12:57 (1781226777) [ 921.537459] LNet: 50577:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 922.124509] Key type .llcrypt unregistered [ 922.125812] Key type ._llcrypt unregistered [ 928.321708] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 928.441730] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 928.445531] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 928.449138] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 930.870452] Key type ._llcrypt registered [ 930.873470] Key type .llcrypt registered [ 931.048202] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 931.082087] alg: No test for adler32 (adler32-zlib) [ 932.041507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 932.146868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 932.178210] LNet: Added LNI 192.168.203.22@tcp [8/512/0/180] [ 932.210174] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 932.212359] LNet: 50904:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 932.384950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 932.405642] LNet: 50923:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 932.410703] LNetError: 50923:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 932.414207] LNet: Removed LNI 192.168.203.22@tcp [ 932.418645] LNetError: 50923:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 932.429743] LNet: Removed LNI 10.1.2.3@tcp [ 932.452233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 932.520327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 932.571320] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 932.664116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 932.676316] LNet: 50947:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 932.679622] LNetError: 50947:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 932.694080] LNet: Removed LNI 192.168.203.22@tcp [ 932.706692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 932.748980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 932.762980] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 932.765753] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 932.767753] LNet: 50955:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 932.819506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 932.839284] LNet: 50973:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 932.845761] LNetError: 50973:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 932.851085] LNet: Removed LNI 192.168.203.22@tcp [ 932.853817] LNetError: 50973:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 932.868839] LNet: Removed LNI 10.1.2.3@tcp [ 932.886750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 932.942216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 932.960576] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 932.963457] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 932.965241] LNet: 50981:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 933.176155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 933.201637] LNet: 51007:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 933.224252] LNetError: 51007:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 933.232844] LNet: Removed LNI 192.168.203.22@tcp [ 933.235646] LNetError: 51007:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 933.252835] LNet: Removed LNI 10.1.2.3@tcp [ 933.274338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 933.337746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 933.361897] LNet: Added LNI 192.168.203.22@tcp [8/256/0/182] [ 933.366576] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 933.496486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 933.528452] LNet: 51035:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 933.537991] LNetError: 51035:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 933.545132] LNet: Removed LNI 192.168.203.22@tcp [ 933.547464] LNetError: 51035:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 933.570671] LNet: Removed LNI 10.1.2.3@tcp [ 933.599690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 933.674574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 933.705481] LNet: Added LNI 192.168.203.22@tcp [8/256/0/182] [ 933.714483] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 933.715852] LNet: 51043:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 933.810896] LNet: Removed LNI 192.168.203.22@tcp [ 933.820554] LNet: Removed LNI 10.1.2.3@tcp [ 934.643168] Key type .llcrypt unregistered [ 934.645954] Key type ._llcrypt unregistered [ 946.881978] Key type ._llcrypt registered [ 946.884791] Key type .llcrypt registered [ 948.188839] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 21:13:24 (1781226804) [ 948.684745] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 948.722029] alg: No test for adler32 (adler32-zlib) [ 949.575221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 949.605814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 949.621481] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 950.586178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.203.22 [ 950.605989] LNet: 52018:0:(config.c:1627:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 950.625947] LNet: Removed LNI 192.168.203.22@tcp [ 957.554463] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 21:13:34 (1781226814) [ 957.721941] LNet: 52582:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 957.738650] LNetError: 52582:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 957.769060] LNet: Removed LNI 192.168.203.22@tcp [ 958.264489] Key type .llcrypt unregistered [ 958.271685] Key type ._llcrypt unregistered [ 964.154557] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 964.414191] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 967.239746] Key type ._llcrypt registered [ 967.246210] Key type .llcrypt registered [ 967.483793] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 967.503572] alg: No test for adler32 (adler32-zlib) [ 968.919199] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 975.588181] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 21:13:52 (1781226832) [ 975.727884] LNet: 53499:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 975.735981] LNetError: 53499:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 975.754190] LNet: Removed LNI 10.1.2.3@tcp [ 976.469712] Key type .llcrypt unregistered [ 976.471439] Key type ._llcrypt unregistered [ 982.268669] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 982.452792] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 982.459550] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 982.466967] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 985.257797] Key type ._llcrypt registered [ 985.259528] Key type .llcrypt registered [ 985.445944] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 985.462383] alg: No test for adler32 (adler32-zlib) [ 986.407065] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 993.158878] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 21:14:09 (1781226849) [ 994.624382] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 996.399573] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 21:14:13 (1781226853) [ 996.687548] LNet: 54686:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 996.697348] LNetError: 54686:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 996.725421] LNet: Removed LNI 10.1.2.3@tcp [ 997.386958] Key type .llcrypt unregistered [ 997.394158] Key type ._llcrypt unregistered [ 1003.634216] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1003.854303] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1003.862142] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1003.869749] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1006.631266] Key type ._llcrypt registered [ 1006.634328] Key type .llcrypt registered [ 1006.863737] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1006.887610] alg: No test for adler32 (adler32-zlib) [ 1007.767609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1007.824580] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1015.174716] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 21:14:32 (1781226872) [ 1015.567604] LNet: 55915:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1015.573547] LNetError: 55915:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1015.597091] LNet: Removed LNI 10.1.2.3@tcp [ 1016.144102] Key type .llcrypt unregistered [ 1016.147075] Key type ._llcrypt unregistered [ 1021.443328] Key type ._llcrypt registered [ 1021.445298] Key type .llcrypt registered [ 1021.676616] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1021.691634] alg: No test for adler32 (adler32-zlib) [ 1022.602512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1022.625817] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1029.479170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.122@tcp [ 1030.208545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.122@tcp -m GET -e local_interrupt -r 1 [ 1030.263207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp -d 192.168.203.22@tcp -m GET -e local_interrupt -r 1 [ 1030.327820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.122@tcp [ 1030.364905] LNet: There was an unexpected network error while writing to 192.168.203.122: rc = -22 [ 1030.598296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1030.630236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1030.665399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1030.693897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1030.739253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1030.765711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1030.788792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1030.816584] LNet: 1 local NIs in recovery (showing 1): 192.168.203.22@tcp [ 1030.818843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1030.844918] LNet: 56352:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1030.864807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1030.912702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1030.948973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1030.990834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1031.016465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1031.055134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1031.096677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1031.582196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp -d 192.168.203.122@tcp -m GET -e local_dropped -r 1 [ 1031.619795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp -d 192.168.203.22@tcp -m GET -e local_dropped -r 1 [ 1031.654658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.122@tcp [ 1031.675995] LNet: There was an unexpected network error while writing to 192.168.203.122: rc = -22 [ 1031.868475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1031.909242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1031.957942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1032.011076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1032.055762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1032.085709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1032.118812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1032.142670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1032.158250] LNet: 56461:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1032.175952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1032.197816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1032.225396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1032.263219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1032.302640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1032.357703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1032.405901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1032.976040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp -d 192.168.203.122@tcp -m GET -e local_aborted -r 1 [ 1033.011838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp -d 192.168.203.22@tcp -m GET -e local_aborted -r 1 [ 1033.062986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.122@tcp [ 1033.093037] LNet: There was an unexpected network error while writing to 192.168.203.122: rc = -22 [ 1033.101224] LNet: Skipped 1 previous similar message [ 1033.298662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1033.331147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1033.354864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1033.392455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1033.415582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1033.436884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1033.460710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1033.489710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1033.507369] LNet: 56571:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1033.524207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1033.548477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1033.567665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1033.589983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1033.613579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1033.644940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1033.684369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1034.255995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.122@tcp -m GET -e local_no_route -r 1 [ 1034.307681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.22@tcp -m GET -e local_no_route -r 1 [ 1034.352831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.122@tcp [ 1034.593259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1034.627543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1034.662371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1034.697495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1034.731179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1034.763830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1034.812388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1034.863522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1034.898690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1034.935445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1034.966434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1035.002236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1035.041499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1035.078631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1035.111961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1035.650306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.122@tcp -m GET -e local_timeout -r 1 [ 1035.703871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp -d 192.168.203.22@tcp -m GET -e local_timeout -r 1 [ 1035.748408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.122@tcp [ 1035.791779] LNet: There was an unexpected network error while writing to 192.168.203.122: rc = -22 [ 1035.799519] LNet: Skipped 1 previous similar message [ 1036.092697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1036.139096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1036.209473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1036.273798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1036.351640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1036.403902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1036.448079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1036.499923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1036.535612] LNet: 56789:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1036.548352] LNet: 56789:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1036.576475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1036.665613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1036.725181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1036.797792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1036.860718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1036.938241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1036.988721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1037.557485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp -d 192.168.203.122@tcp -m GET -e local_error -r 1 [ 1037.593935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.22@tcp -m GET -e local_error -r 1 [ 1037.633769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.122@tcp [ 1037.903618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1037.926222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1037.959515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1037.982995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1038.004344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1038.044687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1038.079414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1038.112214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1038.135805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1038.169846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1038.191451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1038.227239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1038.273720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1038.325805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1038.390531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1038.778919] LNet: 56939:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1038.788359] LNetError: 56939:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1039.849431] LNet: Removed LNI 192.168.203.22@tcp [ 1040.291222] Key type .llcrypt unregistered [ 1040.293990] Key type ._llcrypt unregistered [ 1058.199770] Key type ._llcrypt registered [ 1058.202516] Key type .llcrypt registered [ 1059.578919] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 21:15:16 (1781226916) [ 1060.451324] Key type .llcrypt unregistered [ 1060.453552] Key type ._llcrypt unregistered [ 1067.216396] Key type ._llcrypt registered [ 1067.218790] Key type .llcrypt registered [ 1067.520797] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1067.536328] alg: No test for adler32 (adler32-zlib) [ 1068.501845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1068.541236] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1076.897707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.122@tcp [ 1089.584509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1089.679584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1089.710427] LNet: Added LNI 192.168.203.22@tcp1 [8/256/0/180] [ 1091.312806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.122@tcp -m GET -e local_interrupt -r 1 [ 1091.354584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp -d 192.168.203.122@tcp1 -m GET -e local_interrupt -r 1 [ 1091.400625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp -d 192.168.203.22@tcp -m GET -e local_interrupt -r 1 [ 1091.473813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.22@tcp1 -m GET -e local_interrupt -r 1 [ 1091.520969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp1 -d 192.168.203.122@tcp -m GET -e local_interrupt -r 1 [ 1091.558800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp1 -d 192.168.203.122@tcp1 -m GET -e local_interrupt -r 1 [ 1091.594485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp1 -d 192.168.203.22@tcp -m GET -e local_interrupt -r 1 [ 1091.634589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp1 -d 192.168.203.22@tcp1 -m GET -e local_interrupt -r 1 [ 1091.702528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.122@tcp [ 1091.714488] LNet: There was an unexpected network error while writing to 192.168.203.122: rc = -22 [ 1091.718403] LNet: 1 local NIs in recovery (showing 1): 192.168.203.22@tcp [ 1091.836593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1091.855761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1091.886852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1091.928910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1091.951183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1091.970859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1091.997893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1092.033598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1092.046411] LNet: 58407:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1092.060610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1092.084593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1092.111672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1092.145631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1092.178726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1092.206472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1092.248403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1092.667813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.122@tcp -m GET -e local_dropped -r 1 [ 1092.687290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.122@tcp1 -m GET -e local_dropped -r 1 [ 1092.709175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.22@tcp -m GET -e local_dropped -r 1 [ 1092.734887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.22@tcp1 -m GET -e local_dropped -r 1 [ 1092.753867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp1 -d 192.168.203.122@tcp -m GET -e local_dropped -r 1 [ 1092.776306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp1 -d 192.168.203.122@tcp1 -m GET -e local_dropped -r 1 [ 1092.797751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp1 -d 192.168.203.22@tcp -m GET -e local_dropped -r 1 [ 1092.828917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp1 -d 192.168.203.22@tcp1 -m GET -e local_dropped -r 1 [ 1092.868535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.122@tcp [ 1092.889299] LNet: There was an unexpected network error while writing to 192.168.203.122: rc = -22 [ 1092.898896] LNet: Skipped 2 previous similar messages [ 1093.154996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1093.175256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1093.197541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1093.215911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1093.232856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1093.279805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1093.328127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1093.378543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1093.395238] LNet: 58528:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1093.414927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1093.494276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1093.551485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1093.604869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1093.668971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1093.703233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1093.755537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1094.211235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp -d 192.168.203.122@tcp -m GET -e local_aborted -r 1 [ 1094.240337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp -d 192.168.203.122@tcp1 -m GET -e local_aborted -r 1 [ 1094.265871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.22@tcp -m GET -e local_aborted -r 1 [ 1094.306532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.22@tcp1 -m GET -e local_aborted -r 1 [ 1094.354923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp1 -d 192.168.203.122@tcp -m GET -e local_aborted -r 1 [ 1094.393508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp1 -d 192.168.203.122@tcp1 -m GET -e local_aborted -r 1 [ 1094.423167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp1 -d 192.168.203.22@tcp -m GET -e local_aborted -r 1 [ 1094.460381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp1 -d 192.168.203.22@tcp1 -m GET -e local_aborted -r 1 [ 1094.519650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.122@tcp [ 1094.560540] LNet: There was an unexpected network error while writing to 192.168.203.122: rc = -22 [ 1094.570509] LNet: Skipped 2 previous similar messages [ 1094.825690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1094.871671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1094.917321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1094.979430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1095.029478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1095.088715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1095.132982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1095.177764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1095.234459] LNet: 58650:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1095.268420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1095.317758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1095.365181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1095.393168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1095.464701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1095.512662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1095.550653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1096.167240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp -d 192.168.203.122@tcp -m GET -e local_no_route -r 1 [ 1096.221873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.122@tcp1 -m GET -e local_no_route -r 1 [ 1096.296895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp -d 192.168.203.22@tcp -m GET -e local_no_route -r 1 [ 1096.364676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.22@tcp1 -m GET -e local_no_route -r 1 [ 1096.428593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp1 -d 192.168.203.122@tcp -m GET -e local_no_route -r 1 [ 1096.513638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp1 -d 192.168.203.122@tcp1 -m GET -e local_no_route -r 1 [ 1096.581242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp1 -d 192.168.203.22@tcp -m GET -e local_no_route -r 1 [ 1096.647454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp1 -d 192.168.203.22@tcp1 -m GET -e local_no_route -r 1 [ 1096.687467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.122@tcp [ 1096.712308] LNet: There was an unexpected network error while writing to 192.168.203.122: rc = -22 [ 1096.717962] LNet: Skipped 2 previous similar messages [ 1097.022933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1097.055213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1097.081439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1097.118665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1097.193553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1097.243604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1097.278414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1097.327531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1097.349591] LNet: 58771:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1097.372979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1097.415241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1097.476358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1097.517892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1097.574345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1097.650120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1097.699367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1098.324821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.122@tcp -m GET -e local_timeout -r 1 [ 1098.397973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp -d 192.168.203.122@tcp1 -m GET -e local_timeout -r 1 [ 1098.462054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.22@tcp -m GET -e local_timeout -r 1 [ 1098.503693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.22@tcp1 -m GET -e local_timeout -r 1 [ 1098.553847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp1 -d 192.168.203.122@tcp -m GET -e local_timeout -r 1 [ 1098.599592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp1 -d 192.168.203.122@tcp1 -m GET -e local_timeout -r 1 [ 1098.653637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp1 -d 192.168.203.22@tcp -m GET -e local_timeout -r 1 [ 1098.700858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp1 -d 192.168.203.22@tcp1 -m GET -e local_timeout -r 1 [ 1098.768956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.122@tcp [ 1099.022555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1099.081558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1099.130964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1099.193357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1099.227792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1099.283389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1099.332327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1099.395355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1099.449348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1099.497204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1099.535215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1099.573351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1099.627105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1099.671343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1099.729652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1100.310612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.122@tcp -m GET -e local_error -r 1 [ 1100.367776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.122@tcp1 -m GET -e local_error -r 1 [ 1100.412854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.22@tcp -m GET -e local_error -r 1 [ 1100.458748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.22@tcp1 -m GET -e local_error -r 1 [ 1100.506964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp1 -d 192.168.203.122@tcp -m GET -e local_error -r 1 [ 1100.550168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp1 -d 192.168.203.122@tcp1 -m GET -e local_error -r 1 [ 1100.590760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp1 -d 192.168.203.22@tcp -m GET -e local_error -r 1 [ 1100.627898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp1 -d 192.168.203.22@tcp1 -m GET -e local_error -r 1 [ 1100.670975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.122@tcp [ 1100.875505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1100.901813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1100.928766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1100.955752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1100.978930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1101.003117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1101.026611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1101.053494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1101.094515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1101.126972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1101.158587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1101.195633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1101.233644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1101.259743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1101.283095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1107.195255] LNet: 59090:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1107.205677] LNetError: 59090:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1107.219735] LNet: Removed LNI 192.168.203.22@tcp [ 1107.235395] LNet: Removed LNI 192.168.203.22@tcp1 [ 1107.787204] Key type .llcrypt unregistered [ 1107.789965] Key type ._llcrypt unregistered [ 1123.983194] Key type ._llcrypt registered [ 1123.985288] Key type .llcrypt registered [ 1125.183378] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 21:16:21 (1781226981) [ 1125.956420] Key type .llcrypt unregistered [ 1125.959465] Key type ._llcrypt unregistered [ 1130.743520] Key type ._llcrypt registered [ 1130.746092] Key type .llcrypt registered [ 1130.924339] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1130.970198] alg: No test for adler32 (adler32-zlib) [ 1131.904405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1131.965969] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1138.861900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.122@tcp [ 1139.468814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.122@tcp -m GET -e remote_dropped -r 1 [ 1139.545373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.22@tcp -m GET -e remote_dropped -r 1 [ 1139.591950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.122@tcp [ 1139.609624] LNet: There was an unexpected network error while writing to 192.168.203.122: rc = -22 [ 1139.795385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1139.857539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1139.904792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1139.931892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1139.971860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1140.011409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1140.048403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1140.081875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1140.105842] LNet: 60339:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1140.129269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1140.160556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1140.188977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1140.218923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1140.252949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1140.304444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1140.332587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1140.818938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.122@tcp -m GET -e remote_error -r 1 [ 1140.843422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.22@tcp -m GET -e remote_error -r 1 [ 1140.867504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.122@tcp [ 1140.892196] LNet: There was an unexpected network error while writing to 192.168.203.122: rc = -22 [ 1141.090441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1141.121811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1141.149899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1141.186274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1141.227255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1141.266489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1141.302854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1141.343538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1141.368322] LNet: 60448:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1141.416051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1141.449066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1141.485694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1141.526651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1141.568626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1141.597861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1141.628818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1142.148030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp -d 192.168.203.122@tcp -m GET -e remote_timeout -r 1 [ 1142.183858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.22@tcp -m GET -e remote_timeout -r 1 [ 1142.212865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.122@tcp [ 1142.239335] LNet: There was an unexpected network error while writing to 192.168.203.122: rc = -22 [ 1142.477595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1142.504986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1142.535612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1142.572205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1142.612201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1142.648307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1142.704577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1142.729869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1142.767543] LNet: 60557:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1142.788972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1142.829425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1142.897243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1142.942601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1142.972955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1143.014024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1143.059740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1143.381393] LNet: 60598:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1143.391123] LNetError: 60598:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1143.419593] LNet: Removed LNI 192.168.203.22@tcp [ 1143.874992] Key type .llcrypt unregistered [ 1143.878709] Key type ._llcrypt unregistered [ 1159.458808] Key type ._llcrypt registered [ 1159.461619] Key type .llcrypt registered [ 1160.785710] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 21:16:57 (1781227017) [ 1161.572319] Key type .llcrypt unregistered [ 1161.576126] Key type ._llcrypt unregistered [ 1167.593574] Key type ._llcrypt registered [ 1167.605472] Key type .llcrypt registered [ 1167.884154] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1167.913492] alg: No test for adler32 (adler32-zlib) [ 1168.907729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1168.926965] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1176.048986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.122@tcp [ 1183.807699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1183.843454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1183.864635] LNet: Added LNI 192.168.203.22@tcp1 [8/256/0/180] [ 1185.066686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp -d 192.168.203.122@tcp -m GET -e remote_dropped -r 1 [ 1185.097946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.122@tcp1 -m GET -e remote_dropped -r 1 [ 1185.151654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp -d 192.168.203.22@tcp -m GET -e remote_dropped -r 1 [ 1185.185059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.22@tcp1 -m GET -e remote_dropped -r 1 [ 1185.241176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp1 -d 192.168.203.122@tcp -m GET -e remote_dropped -r 1 [ 1185.284443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp1 -d 192.168.203.122@tcp1 -m GET -e remote_dropped -r 1 [ 1185.317708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp1 -d 192.168.203.22@tcp -m GET -e remote_dropped -r 1 [ 1185.350524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp1 -d 192.168.203.22@tcp1 -m GET -e remote_dropped -r 1 [ 1185.379637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.122@tcp [ 1185.395443] LNet: There was an unexpected network error while writing to 192.168.203.122: rc = -22 [ 1185.400177] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.122@tcp [ 1185.629253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1185.667716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1185.699242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1185.724659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1185.767250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1185.808816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1185.851460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1185.891447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1185.915297] LNet: 62064:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1185.943398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1186.000846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1186.030450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1186.074592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1186.106887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1186.136713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1186.161985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1186.193868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1186.783583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.122@tcp -m GET -e remote_error -r 1 [ 1186.829253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.122@tcp1 -m GET -e remote_error -r 1 [ 1186.874722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp -d 192.168.203.22@tcp -m GET -e remote_error -r 1 [ 1186.909916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.22@tcp1 -m GET -e remote_error -r 1 [ 1186.938200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp1 -d 192.168.203.122@tcp -m GET -e remote_error -r 1 [ 1186.979641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp1 -d 192.168.203.122@tcp1 -m GET -e remote_error -r 1 [ 1187.022375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp1 -d 192.168.203.22@tcp -m GET -e remote_error -r 1 [ 1187.059560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp1 -d 192.168.203.22@tcp1 -m GET -e remote_error -r 1 [ 1187.095854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.122@tcp [ 1187.112349] LNet: There was an unexpected network error while writing to 192.168.203.122: rc = -22 [ 1187.119503] LNet: Skipped 2 previous similar messages [ 1187.360647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1187.395690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1187.423583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1187.453484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1187.472540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1187.493947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1187.523753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1187.562610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1187.579460] LNet: 62188:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1187.598732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1187.627963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1187.661931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1187.715052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1187.743394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1187.777891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1187.810258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1187.841160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1188.290654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.122@tcp -m GET -e remote_timeout -r 1 [ 1188.336259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp -d 192.168.203.122@tcp1 -m GET -e remote_timeout -r 1 [ 1188.378617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.22@tcp -m GET -e remote_timeout -r 1 [ 1188.430645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.22@tcp1 -m GET -e remote_timeout -r 1 [ 1188.461299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp1 -d 192.168.203.122@tcp -m GET -e remote_timeout -r 1 [ 1188.498917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp1 -d 192.168.203.122@tcp1 -m GET -e remote_timeout -r 1 [ 1188.536264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp1 -d 192.168.203.22@tcp -m GET -e remote_timeout -r 1 [ 1188.563040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp1 -d 192.168.203.22@tcp1 -m GET -e remote_timeout -r 1 [ 1188.590652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.122@tcp [ 1188.624237] LNet: There was an unexpected network error while writing to 192.168.203.122: rc = -22 [ 1188.821524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1188.864066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1188.916182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1188.958290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1188.992159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1189.023361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1189.056972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1189.094835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1189.129420] LNet: 62311:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1189.156785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1189.205515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1189.283461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1189.332788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1189.376366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1189.424366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1189.460517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1189.497765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1193.975758] LNet: 62388:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1193.983088] LNetError: 62388:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1193.993847] LNet: Removed LNI 192.168.203.22@tcp [ 1194.020070] LNet: Removed LNI 192.168.203.22@tcp1 [ 1194.686447] Key type .llcrypt unregistered [ 1194.688104] Key type ._llcrypt unregistered [ 1213.322749] Key type ._llcrypt registered [ 1213.324548] Key type .llcrypt registered [ 1214.528823] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 21:17:51 (1781227071) [ 1215.902599] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1217.320982] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 21:17:54 (1781227074) [ 1218.081768] Key type .llcrypt unregistered [ 1218.086912] Key type ._llcrypt unregistered [ 1225.022109] Key type ._llcrypt registered [ 1225.029102] Key type .llcrypt registered [ 1225.241951] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1225.271277] alg: No test for adler32 (adler32-zlib) [ 1226.375789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1226.418169] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1235.145602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.122@tcp [ 1235.740338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.122@tcp -m GET -e network_timeout -r 1 [ 1235.769618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.22@tcp -m GET -e network_timeout -r 1 [ 1235.801572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.122@tcp [ 1235.816876] LNet: There was an unexpected network error while writing to 192.168.203.122: rc = -22 [ 1236.123425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1236.210598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1236.286656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1236.347468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1236.423602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1236.478718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1236.528625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1236.564429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1236.583288] LNet: 63933:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1236.601268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1236.641179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1236.678452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1236.705595] LNet: 1 local NIs in recovery (showing 1): 192.168.203.22@tcp [ 1236.707138] LNet: There was an unexpected network error while writing to 192.168.203.22: rc = -22 [ 1236.711032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1236.751199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1236.816989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1237.115615] LNet: 63972:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1237.123414] LNetError: 63972:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1237.143682] LNet: Removed LNI 192.168.203.22@tcp [ 1237.618294] Key type .llcrypt unregistered [ 1237.621234] Key type ._llcrypt unregistered [ 1255.753288] Key type ._llcrypt registered [ 1255.764862] Key type .llcrypt registered [ 1256.018039] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1256.025707] alg: No test for adler32 (adler32-zlib) [ 1256.865860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1256.880630] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1265.107944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.122@tcp [ 1275.107729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1275.159338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1275.183554] LNet: Added LNI 192.168.203.22@tcp1 [8/256/0/180] [ 1276.562289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp -d 192.168.203.122@tcp -m GET -e network_timeout -r 1 [ 1276.593247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp -d 192.168.203.122@tcp1 -m GET -e network_timeout -r 1 [ 1276.629242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp -d 192.168.203.22@tcp -m GET -e network_timeout -r 1 [ 1276.668962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp -d 192.168.203.22@tcp1 -m GET -e network_timeout -r 1 [ 1276.748171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp1 -d 192.168.203.122@tcp -m GET -e network_timeout -r 1 [ 1276.833146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp1 -d 192.168.203.122@tcp1 -m GET -e network_timeout -r 1 [ 1276.893251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp1 -d 192.168.203.22@tcp -m GET -e network_timeout -r 1 [ 1276.926885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp1 -d 192.168.203.22@tcp1 -m GET -e network_timeout -r 1 [ 1276.953293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.122@tcp [ 1276.979845] LNet: There was an unexpected network error while writing to 192.168.203.122: rc = -22 [ 1277.208576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1277.245453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1277.272331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1277.294822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1277.338207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1277.344398] LNet: 1 local NIs in recovery (showing 1): 192.168.203.22@tcp [ 1277.357548] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.122@tcp [ 1277.388854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1277.439335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1277.474698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1277.518808] LNet: 64851:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1277.543111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1277.583614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1277.620835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1277.651815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1277.693921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1277.730277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1282.158932] LNet: There was an unexpected network error while writing to 192.168.203.122: rc = -22 [ 1282.167301] LNet: Skipped 2 previous similar messages [ 1282.488689] LNet: 64925:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1282.495267] LNetError: 64925:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1282.509944] LNet: Removed LNI 192.168.203.22@tcp [ 1282.529174] LNet: Removed LNI 192.168.203.22@tcp1 [ 1282.927733] Key type .llcrypt unregistered [ 1282.933431] Key type ._llcrypt unregistered [ 1299.032669] Key type ._llcrypt registered [ 1299.034949] Key type .llcrypt registered [ 1300.323510] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 21:19:17 (1781227157) [ 1300.705872] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1300.722685] alg: No test for adler32 (adler32-zlib) [ 1301.649487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1301.701644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1301.717767] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1302.461696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1302.484706] LNet: Added LNI 192.168.203.22@tcp1 [8/256/0/180] [ 1303.454401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.22@tcp [ 1303.565537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1303.610679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1303.640805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1303.683959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1303.739811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1303.801689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.22@tcp [ 1303.833837] LNetError: 66025:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.203.22@tcp to recovery [ 1304.736337] LNet: 1 local NIs in recovery (showing 1): 192.168.203.22@tcp [ 1304.736612] LNet: There was an unexpected network error while writing to 192.168.203.22: rc = -22 [ 1306.784847] LNet: There was an unexpected network error while writing to 192.168.203.22: rc = -22 [ 1307.429478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1310.883489] LNet: There was an unexpected network error while writing to 192.168.203.22: rc = -22 [ 1312.144055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1312.180284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1312.259335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1312.291657] LNet: 66143:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1312.305779] LNetError: 66143:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1313.376147] LNet: Removed LNI 192.168.203.22@tcp [ 1313.381760] LNetError: 66143:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1313.401720] LNet: Removed LNI 192.168.203.22@tcp1 [ 1313.435906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1313.543185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1313.577214] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1314.273631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1314.293457] LNet: Added LNI 192.168.203.22@tcp1 [8/256/0/180] [ 1315.186416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.22@tcp [ 1315.231844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1315.303499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1315.343379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1315.395261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1315.444661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1315.481987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1315.514646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.22@tcp [ 1315.533874] LNetError: 66415:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.203.22@tcp to recovery [ 1315.555815] LNet: There was an unexpected network error while writing to 192.168.203.22: rc = -22 [ 1318.036357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1321.696356] LNet: There was an unexpected network error while writing to 192.168.203.22: rc = -22 [ 1321.704544] LNet: Skipped 1 previous similar message [ 1326.028201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1326.081457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1326.119700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1332.877935] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 21:19:49 (1781227189) [ 1332.981246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1332.999835] LNet: 67117:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1333.005321] LNetError: 67117:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1333.010199] LNet: Removed LNI 192.168.203.22@tcp [ 1333.021132] LNet: Removed LNI 192.168.203.22@tcp1 [ 1333.066831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1333.116568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1333.135885] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1333.955185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1334.006742] LNet: Added LNI 192.168.203.22@tcp1 [8/256/0/180] [ 1335.071209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.22@tcp [ 1335.148156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1335.176758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1335.206721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1335.244959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1335.278805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1335.331744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.22@tcp --health 0 [ 1335.444702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1336.160504] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.22@tcp [ 1336.164382] LNet: There was an unexpected network error while writing to 192.168.203.22: rc = -22 [ 1336.182774] LNet: Skipped 1 previous similar message [ 1350.829099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1351.007946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1351.055433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1351.100846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.22@tcp --health 500 [ 1351.226741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1354.623848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1354.672372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1354.685680] LNet: 67590:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1354.696398] LNetError: 67590:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1354.709870] LNetError: 67590:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1354.717701] LNet: Removed LNI 192.168.203.22@tcp [ 1354.727443] LNet: Removed LNI 192.168.203.22@tcp1 [ 1354.749853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1354.791470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1354.809573] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1355.579634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1355.624083] LNet: Added LNI 192.168.203.22@tcp1 [8/256/0/180] [ 1356.645680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.22@tcp [ 1356.692622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1356.754934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1356.786663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1356.828288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1356.865785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1356.900406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1356.930963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.22@tcp --health 0 [ 1357.864425] LNet: There was an unexpected network error while writing to 192.168.203.22: rc = -22 [ 1357.875601] LNet: Skipped 3 previous similar messages [ 1358.313606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1368.698483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1368.743196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1368.804931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1377.169430] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 21:20:33 (1781227233) [ 1377.752786] LNet: 68636:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1377.768963] LNetError: 68636:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1377.775759] LNetError: 68636:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1377.783104] LNet: Removed LNI 192.168.203.22@tcp [ 1377.795047] LNet: Removed LNI 192.168.203.22@tcp1 [ 1378.561815] Key type .llcrypt unregistered [ 1378.568516] Key type ._llcrypt unregistered [ 1383.589504] Key type ._llcrypt registered [ 1383.591268] Key type .llcrypt registered [ 1383.781717] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1383.809229] alg: No test for adler32 (adler32-zlib) [ 1384.833947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1384.867786] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1394.299980] Lustre: DEBUG MARKER: Initial discovery [ 1394.401027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.122@tcp [ 1396.350681] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1396.513466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1396.541519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.122@tcp [ 1396.578572] LNet: There was an unexpected network error while writing to 192.168.203.122: rc = -22 [ 1396.626855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1396.691509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.203.122@tcp -r 1 -m GET -l 3 [ 1396.755238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.203.122@tcp -r 1 -m GET -e local_error [ 1396.834380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.203.122@tcp -r 1 -m PUT -l 6 [ 1397.157834] LNet: 1 local NIs in recovery (showing 1): 192.168.203.22@tcp [ 1398.375988] Lustre: DEBUG MARKER: Force oleg322-client.virtnet to discover 192.168.203.122@tcp (in background) [ 1398.616738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.122@tcp [ 1400.026268] Lustre: DEBUG MARKER: Force oleg322-server to discover 192.168.203.22@tcp [ 1401.701398] LNet: There was an unexpected network error while writing to 192.168.203.122: rc = -22 [ 1402.037853] Lustre: DEBUG MARKER: Wait for 69399 [ 1403.373886] Lustre: DEBUG MARKER: Finished wait on 69399 [ 1403.503795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1403.543755] LNet: There was an unexpected network error while writing to 192.168.203.122: rc = -22 [ 1403.549987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1403.790394] LNet: 69849:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1403.797727] LNetError: 69849:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1403.813824] LNet: Removed LNI 192.168.203.22@tcp [ 1404.113977] Key type .llcrypt unregistered [ 1404.116275] Key type ._llcrypt unregistered [ 1420.327675] Key type ._llcrypt registered [ 1420.331576] Key type .llcrypt registered [ 1421.601898] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 21:21:18 (1781227278) [ 1422.191843] Key type .llcrypt unregistered [ 1422.195307] Key type ._llcrypt unregistered [ 1428.176302] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1428.298544] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1428.306780] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1428.317556] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1430.924705] Key type ._llcrypt registered [ 1430.930741] Key type .llcrypt registered [ 1431.094865] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1431.117300] alg: No test for adler32 (adler32-zlib) [ 1432.029358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1432.091550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1432.134109] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1432.815679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1432.834857] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1440.427377] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 21:21:37 (1781227297) [ 1440.557254] LNet: 71921:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1440.564784] LNetError: 71921:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1440.580192] LNet: Removed LNI 192.168.203.22@tcp [ 1440.611131] LNet: Removed LNI 10.1.2.3@tcp [ 1441.468485] Key type .llcrypt unregistered [ 1441.470090] Key type ._llcrypt unregistered [ 1448.998216] Key type ._llcrypt registered [ 1448.999679] Key type .llcrypt registered [ 1449.199549] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1449.219653] alg: No test for adler32 (adler32-zlib) [ 1450.158769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1450.232298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1450.257167] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1450.975666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1450.992073] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1451.945026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.203.22@tcp [ 1453.030126] LNetError: Unexpected error -22 connecting to 192.168.203.22@tcp at host 192.168.203.22:988 [ 1458.513978] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 21:21:55 (1781227315) [ 1458.837175] LNet: 73058:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1458.851298] LNetError: 73058:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1459.872162] LNet: Removed LNI 192.168.203.22@tcp [ 1459.874975] LNetError: 73058:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1459.899806] LNet: Removed LNI 10.1.2.3@tcp [ 1460.590434] Key type .llcrypt unregistered [ 1460.592200] Key type ._llcrypt unregistered [ 1466.589378] Key type ._llcrypt registered [ 1466.592288] Key type .llcrypt registered [ 1466.834726] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1466.854875] alg: No test for adler32 (adler32-zlib) [ 1467.841349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1467.919299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1467.966443] LNet: Added LNI 192.168.203.22@tcp1 [8/256/0/180] [ 1468.881734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1468.905662] LNet: Added LNI 192.168.203.22@tcp2 [8/256/0/180] [ 1469.856959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.203.22@tcp1 --nid 192.168.203.22@tcp2 [ 1474.417430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.22@tcp1 192.168.203.22@tcp2 [ 1474.432390] LNetError: 73894:0:(lib-move.c:2007:lnet_handle_find_routed_path()) Unable to send message from 192.168.203.22@tcp1 to 192.168.203.22@tcp2 - Route table may be misconfigured [ 1474.458579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.22@tcp2 192.168.203.22@tcp1 [ 1480.226607] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 21:22:17 (1781227337) [ 1480.294531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1480.320442] LNet: 74461:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1480.326805] LNetError: 74461:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1480.333274] LNet: Removed LNI 192.168.203.22@tcp1 [ 1481.397429] LNet: Removed LNI 192.168.203.22@tcp2 [ 1481.428738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1481.506973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1481.533606] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1482.136160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1482.161496] LNet: Added LNI 192.168.203.22@tcp1 [8/256/0/180] [ 1482.877451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.22@tcp [ 1482.907477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.22@tcp 192.168.203.22@tcp [ 1482.951641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.22@tcp1 192.168.203.22@tcp1 [ 1483.016350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.22@tcp -d 192.168.203.22@tcp -e network_timeout [ 1483.061914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.22@tcp -d 192.168.203.22@tcp1 -e network_timeout [ 1483.117412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.22@tcp1 -d 192.168.203.22@tcp -e network_timeout [ 1483.148659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.22@tcp1 -d 192.168.203.22@tcp1 -e network_timeout [ 1483.196334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.22@tcp [ 1483.218336] LNet: There was an unexpected network error while writing to 192.168.203.22: rc = -22 [ 1483.242662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1483.341842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1483.471193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1483.488335] LNet: 1 local NIs in recovery (showing 1): 192.168.203.22@tcp [ 1488.233629] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 21:22:25 (1781227345) [ 1488.432874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1488.472518] LNet: 75328:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1488.483513] LNetError: 75328:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1488.492840] LNetError: 75328:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1488.500193] LNet: Removed LNI 192.168.203.22@tcp [ 1489.574522] LNet: Removed LNI 192.168.203.22@tcp1 [ 1489.614575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1489.689385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1489.940373] LNet: 75365:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1490.246913] Key type .llcrypt unregistered [ 1490.249355] Key type ._llcrypt unregistered [ 1499.007596] Key type ._llcrypt registered [ 1499.011132] Key type .llcrypt registered [ 1499.954690] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 21:22:37 (1781227357) [ 1500.282218] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1500.299390] alg: No test for adler32 (adler32-zlib) [ 1501.190288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1502.350296] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1503.701307] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 21:22:40 (1781227360) [ 1503.890076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1503.907281] LNet: 76463:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1503.928125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1503.991669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1504.028248] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1504.557534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1504.588994] LNet: Added LNI 192.168.203.22@tcp1 [8/256/0/180] [ 1505.277900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.22@tcp [ 1505.316903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.22@tcp1 [ 1505.358390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.22@tcp1 [ 1510.014549] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 21:22:47 (1781227367) [ 1511.339364] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg322-server [ 1512.592663] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 21:22:49 (1781227369) [ 1514.082114] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg322-server [ 1515.441025] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 21:22:52 (1781227372) [ 1516.779419] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg322-server [ 1518.561336] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 21:22:55 (1781227375) [ 1520.231357] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg322-server [ 1521.579224] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 21:22:58 (1781227378) [ 1523.086942] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg322-server [ 1524.508153] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 21:23:01 (1781227381) [ 1526.302571] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg322-server [ 1527.553147] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 21:23:04 (1781227384) [ 1529.099768] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg322-server [ 1530.392669] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 21:23:07 (1781227387) [ 1531.895949] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg322-server [ 1533.377489] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 21:23:10 (1781227390) [ 1534.876118] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg322-server [ 1536.351459] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 21:23:13 (1781227393) [ 1536.497104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1536.526971] LNet: 80338:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1536.544754] LNetError: 80338:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1536.554741] LNet: Removed LNI 192.168.203.22@tcp [ 1537.584536] LNet: Removed LNI 192.168.203.22@tcp1 [ 1537.619810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1537.702863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1537.732968] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1538.387610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1538.454842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.22@tcp [ 1538.639994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1538.661920] LNet: 80486:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1538.674589] LNetError: 80486:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1538.680875] LNetError: 80486:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1538.693759] LNet: Removed LNI 192.168.203.22@tcp [ 1538.710677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1538.747985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1538.769659] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1539.388866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.22@tcp [ 1539.551388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1540.647404] LNet: Removed LNI 192.168.203.22@tcp [ 1540.680415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1540.772258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1540.803185] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1541.523562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1541.613264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.22@tcp [ 1541.758642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1541.785895] LNet: 80779:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1541.800181] LNet: 80779:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1541.813394] LNetError: 80779:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1541.824758] LNetError: 80779:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1541.844793] LNet: Removed LNI 192.168.203.22@tcp [ 1541.857517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1541.902825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1541.934345] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1542.512426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.22@tcp [ 1542.746841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1543.785662] LNet: Removed LNI 192.168.203.22@tcp [ 1543.822936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1543.866306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1543.882072] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1544.441051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1544.541246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.22@tcp [ 1544.728912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1544.743490] LNetError: 81071:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1544.747695] LNetError: 81071:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1544.759065] LNet: Removed LNI 192.168.203.22@tcp [ 1544.778395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1544.837566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1544.869915] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1545.484776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.22@tcp [ 1545.658521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1546.728784] LNet: Removed LNI 192.168.203.22@tcp [ 1546.759856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1546.807502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1546.824863] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1547.540550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1547.617865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.22@tcp [ 1547.865920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1547.881359] LNet: 81364:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1547.888285] LNet: 81364:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1548.966693] LNet: Removed LNI 192.168.203.22@tcp [ 1548.992898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1549.080674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1549.137506] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1550.038415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.22@tcp [ 1551.488967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1551.509084] LNetError: 81518:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1551.517990] LNetError: 81518:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1552.551545] LNet: Removed LNI 192.168.203.22@tcp [ 1552.571702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1552.633249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1552.660957] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1553.273422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1553.342563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.22@tcp [ 1553.550833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1554.601328] LNet: Removed LNI 192.168.203.22@tcp [ 1554.665435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1554.738910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1554.800783] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1555.608398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.22@tcp [ 1555.925945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1555.989263] LNet: 81811:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1555.995948] LNet: 81811:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1557.039131] LNet: Removed LNI 192.168.203.22@tcp [ 1557.082843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1557.143508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1557.174157] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1557.985995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1558.083656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.22@tcp [ 1558.334994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1559.398642] LNet: Removed LNI 192.168.203.22@tcp [ 1559.442372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1559.497535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1559.525736] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1560.358190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.22@tcp [ 1561.835655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1561.862984] LNetError: 82113:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1561.872803] LNetError: 82113:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1561.892176] LNet: Removed LNI 192.168.203.22@tcp [ 1561.913107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1561.967389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1561.985919] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1562.703653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1562.769375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1562.824886] LNet: Removed LNI 192.168.203.22@tcp [ 1562.848844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1562.896259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1562.914969] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1563.632446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1570.305972] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 21:23:47 (1781227427) [ 1570.385992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1570.429928] LNet: Removed LNI 192.168.203.22@tcp [ 1570.468764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1570.504665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1570.532303] LNet: Added LNI 192.168.203.22@tcp231 [8/256/0/180] [ 1571.780836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1571.813688] LNet: Removed LNI 192.168.203.22@tcp231 [ 1571.831664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1571.899853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1571.916521] LNet: Added LNI 192.168.203.22@tcp231 [8/256/0/180] [ 1571.920515] LNet: 83095:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1572.291075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1572.328843] LNet: Removed LNI 192.168.203.22@tcp231 [ 1572.353772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1572.383946] LNet: Added LNI 192.168.203.22@tcp231 [8/256/0/180] [ 1573.437890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1573.464294] LNet: 83239:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1573.475206] LNet: 83239:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 1573.502632] LNet: Removed LNI 192.168.203.22@tcp231 [ 1573.551904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1573.616996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1573.647490] LNet: Added LNI 192.168.203.22@tcp231 [8/256/0/0] [ 1573.657934] LNet: 83247:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1574.077398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1574.101699] LNet: Removed LNI 192.168.203.22@tcp231 [ 1574.120458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1574.156392] LNet: Added LNI 192.168.203.22@tcp231 [8/256/0/0] [ 1575.219404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1575.259964] LNet: Removed LNI 192.168.203.22@tcp231 [ 1575.285857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1575.352716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1575.372496] LNet: Added LNI 192.168.203.22@tcp231 [8/256/0/60] [ 1575.377061] LNet: 83399:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1575.650806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1575.686959] LNet: Removed LNI 192.168.203.22@tcp231 [ 1575.707092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1575.732884] LNet: Added LNI 192.168.203.22@tcp231 [8/256/0/60] [ 1576.973763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1577.051822] LNet: Removed LNI 192.168.203.22@tcp231 [ 1577.087953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1577.232766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1577.285960] LNet: Added LNI 192.168.203.22@tcp231 [8/256/0/180] [ 1577.724414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1577.770677] LNet: Removed LNI 192.168.203.22@tcp231 [ 1577.798971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1577.840486] LNet: Added LNI 192.168.203.22@tcp231 [8/256/0/180] [ 1584.849126] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 21:24:01 (1781227441) [ 1584.955217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1584.997597] LNetError: 84256:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1585.006525] LNetError: 84256:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1585.035777] LNet: Removed LNI 192.168.203.22@tcp231 [ 1585.065578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1585.115554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1585.135554] LNet: Added LNI 192.168.203.22@tcp232 [8/256/0/180] [ 1585.750419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1585.805857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1585.848948] LNet: Removed LNI 192.168.203.22@tcp232 [ 1585.864833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1585.900655] LNet: Added LNI 192.168.203.22@tcp232 [8/256/0/180] [ 1586.554773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1586.609254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1586.641495] LNet: Removed LNI 192.168.203.22@tcp232 [ 1586.658091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1586.681103] LNet: Added LNI 192.168.203.22@tcp232 [8/256/0/180] [ 1587.343503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1587.377398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1587.410817] LNet: Removed LNI 192.168.203.22@tcp232 [ 1587.437592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1587.472518] LNet: Added LNI 192.168.203.22@tcp232 [8/256/0/180] [ 1588.195222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1588.229123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1588.271859] LNet: Removed LNI 192.168.203.22@tcp232 [ 1588.294875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1594.383979] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 21:24:11 (1781227451) [ 1595.384441] Key type .llcrypt unregistered [ 1595.386873] Key type ._llcrypt unregistered [ 1602.396297] Key type ._llcrypt registered [ 1602.398756] Key type .llcrypt registered [ 1602.542899] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1602.558361] alg: No test for adler32 (adler32-zlib) [ 1603.490266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1603.523787] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1611.376454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.122@tcp [ 1621.260801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1621.313750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1621.339507] LNet: Added LNI 192.168.203.22@tcp1 [8/256/0/180] [ 1623.021558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.122@tcp -m GET -e local_interrupt -r 2 [ 1623.089187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.122@tcp1 -m GET -e local_interrupt -r 2 [ 1623.144602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.22@tcp -m GET -e local_interrupt -r 2 [ 1623.214827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.22@tcp1 -m GET -e local_interrupt -r 2 [ 1623.246117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp1 -d 192.168.203.122@tcp -m GET -e local_interrupt -r 2 [ 1623.295326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp1 -d 192.168.203.122@tcp1 -m GET -e local_interrupt -r 2 [ 1623.359366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp1 -d 192.168.203.22@tcp -m GET -e local_interrupt -r 2 [ 1623.454959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp1 -d 192.168.203.22@tcp1 -m GET -e local_interrupt -r 2 [ 1623.515439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.22@tcp 192.168.203.122@tcp [ 1623.566937] LNet: There was an unexpected network error while writing to 192.168.203.122: rc = -22 [ 1623.574069] LNet: 1 local NIs in recovery (showing 1): 192.168.203.22@tcp [ 1623.912685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1623.943673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1623.986881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1624.021488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1624.051725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1624.079903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1624.107687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1624.135574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1624.155718] LNet: 86050:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1624.176586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1624.208765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1624.236383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1624.259495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1624.293483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1624.321618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1624.852506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.122@tcp -m GET -e local_dropped -r 2 [ 1624.887062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.122@tcp1 -m GET -e local_dropped -r 2 [ 1624.923981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.22@tcp -m GET -e local_dropped -r 2 [ 1624.964582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.22@tcp1 -m GET -e local_dropped -r 2 [ 1625.010789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp1 -d 192.168.203.122@tcp -m GET -e local_dropped -r 2 [ 1625.054226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp1 -d 192.168.203.122@tcp1 -m GET -e local_dropped -r 2 [ 1625.114631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp1 -d 192.168.203.22@tcp -m GET -e local_dropped -r 2 [ 1625.179189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp1 -d 192.168.203.22@tcp1 -m GET -e local_dropped -r 2 [ 1625.258921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.22@tcp 192.168.203.122@tcp [ 1625.285762] LNet: There was an unexpected network error while writing to 192.168.203.122: rc = -22 [ 1625.554122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1625.585591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1625.610656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1625.650830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1625.702307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1625.756325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1625.795773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1625.831858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1625.870786] LNet: 86170:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1625.894693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1625.949250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1625.995022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1626.039600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1626.093543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1626.131286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1626.775760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp -d 192.168.203.122@tcp -m GET -e local_aborted -r 2 [ 1626.828381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.122@tcp1 -m GET -e local_aborted -r 2 [ 1626.871210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp -d 192.168.203.22@tcp -m GET -e local_aborted -r 2 [ 1626.913711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.22@tcp1 -m GET -e local_aborted -r 2 [ 1626.951600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp1 -d 192.168.203.122@tcp -m GET -e local_aborted -r 2 [ 1626.998222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp1 -d 192.168.203.122@tcp1 -m GET -e local_aborted -r 2 [ 1627.040386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp1 -d 192.168.203.22@tcp -m GET -e local_aborted -r 2 [ 1627.075633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp1 -d 192.168.203.22@tcp1 -m GET -e local_aborted -r 2 [ 1627.110514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.22@tcp 192.168.203.122@tcp [ 1627.135759] LNet: There was an unexpected network error while writing to 192.168.203.122: rc = -22 [ 1627.347733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1627.379359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1627.411573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1627.440408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1627.487344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1627.531388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1627.570895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1627.604506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1627.628676] LNet: 86290:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1627.651618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1627.686037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1627.723942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1627.780434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1627.817137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1627.871913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1628.473253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp -d 192.168.203.122@tcp -m GET -e local_no_route -r 2 [ 1628.510311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.122@tcp1 -m GET -e local_no_route -r 2 [ 1628.534489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp -d 192.168.203.22@tcp -m GET -e local_no_route -r 2 [ 1628.561164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.22@tcp1 -m GET -e local_no_route -r 2 [ 1628.586823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp1 -d 192.168.203.122@tcp -m GET -e local_no_route -r 2 [ 1628.611384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp1 -d 192.168.203.122@tcp1 -m GET -e local_no_route -r 2 [ 1628.648126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp1 -d 192.168.203.22@tcp -m GET -e local_no_route -r 2 [ 1628.691689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp1 -d 192.168.203.22@tcp1 -m GET -e local_no_route -r 2 [ 1628.754864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.22@tcp 192.168.203.122@tcp [ 1629.086484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1629.121522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1629.160395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1629.193581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1629.232751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1629.273157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1629.307559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1629.362161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1629.401288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1629.427942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1629.460199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1629.489702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1629.513574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1629.546334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1630.134267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.122@tcp -m GET -e local_timeout -r 2 [ 1630.201458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.122@tcp1 -m GET -e local_timeout -r 2 [ 1630.257991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.22@tcp -m GET -e local_timeout -r 2 [ 1630.303620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.22@tcp1 -m GET -e local_timeout -r 2 [ 1630.361064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp1 -d 192.168.203.122@tcp -m GET -e local_timeout -r 2 [ 1630.432867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp1 -d 192.168.203.122@tcp1 -m GET -e local_timeout -r 2 [ 1630.475292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp1 -d 192.168.203.22@tcp -m GET -e local_timeout -r 2 [ 1630.528892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp1 -d 192.168.203.22@tcp1 -m GET -e local_timeout -r 2 [ 1630.585508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.22@tcp 192.168.203.122@tcp [ 1630.634723] LNet: There was an unexpected network error while writing to 192.168.203.122: rc = -22 [ 1630.641613] LNet: Skipped 1 previous similar message [ 1630.906099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1630.937945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1630.973687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1631.003231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1631.037951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1631.073197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1631.099267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1631.137547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1631.161779] LNet: 86531:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1631.172921] LNet: 86531:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1631.213175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1631.259329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1631.306087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1631.388343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1631.437947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1631.498432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1632.044401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp -d 192.168.203.122@tcp -m GET -e remote_dropped -r 2 [ 1632.077728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.122@tcp1 -m GET -e remote_dropped -r 2 [ 1632.106860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.22@tcp -m GET -e remote_dropped -r 2 [ 1632.133337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp -d 192.168.203.22@tcp1 -m GET -e remote_dropped -r 2 [ 1632.172853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp1 -d 192.168.203.122@tcp -m GET -e remote_dropped -r 2 [ 1632.212802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp1 -d 192.168.203.122@tcp1 -m GET -e remote_dropped -r 2 [ 1632.241306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp1 -d 192.168.203.22@tcp -m GET -e remote_dropped -r 2 [ 1632.281202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp1 -d 192.168.203.22@tcp1 -m GET -e remote_dropped -r 2 [ 1632.347740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.22@tcp 192.168.203.122@tcp [ 1632.378580] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.122@tcp [ 1632.583346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1632.622454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1632.675290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1632.717678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1632.752950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1632.777605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1632.806836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1632.831671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1632.859517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1632.892677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1632.930981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1632.978337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1633.038165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1633.086530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1639.241723] LNet: 86725:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1639.251189] LNetError: 86725:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1639.264474] LNet: Removed LNI 192.168.203.22@tcp [ 1639.279820] LNet: Removed LNI 192.168.203.22@tcp1 [ 1639.714587] Key type .llcrypt unregistered [ 1639.721857] Key type ._llcrypt unregistered [ 1655.633323] Key type ._llcrypt registered [ 1655.642543] Key type .llcrypt registered [ 1656.800539] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 21:25:13 (1781227513) [ 1657.104111] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1657.120146] alg: No test for adler32 (adler32-zlib) [ 1658.099569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1658.157928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1658.181165] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1659.167747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1659.208593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.22@tcp [ 1659.225150] LNetError: 87703:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.203.22@tcp to recovery [ 1659.310867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.203.22@tcp [ 1659.422402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1666.025362] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 21:25:22 (1781227522) [ 1666.148614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1666.191423] LNet: 88288:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1666.205246] LNetError: 88288:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1666.231842] LNet: Removed LNI 192.168.203.22@tcp [ 1666.273851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1666.315593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1666.339670] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1666.971587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1666.992991] LNet: Added LNI 192.168.203.22@tcp1 [8/256/0/180] [ 1667.947208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.203.22@tcp [ 1667.981849] LNetError: 88558:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.203.22@tcp to recovery [ 1668.320328] LNet: 1 local NIs in recovery (showing 1): 192.168.203.22@tcp [ 1668.321879] LNet: There was an unexpected network error while writing to 192.168.203.22: rc = -22 [ 1669.175900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.203.22@tcp [ 1677.684460] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 21:25:34 (1781227534) [ 1679.397082] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg322-server [ 1681.158545] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 21:25:37 (1781227537) [ 1682.811070] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg322-server [ 1684.433635] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 21:25:41 (1781227541) [ 1686.493452] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg322-server [ 1688.288877] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 21:25:45 (1781227545) [ 1688.502438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1688.534242] LNet: 90189:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1688.541044] LNetError: 90189:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1688.545768] LNet: Removed LNI 192.168.203.22@tcp [ 1688.575312] LNet: Removed LNI 192.168.203.22@tcp1 [ 1688.617931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1688.701819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1688.747299] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1689.555367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1689.624239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1689.671429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1689.741452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1689.782928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1689.819776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1695.717678] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 21:25:52 (1781227552) [ 1695.892831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1695.925562] LNet: 90916:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1695.931123] LNetError: 90916:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1695.934918] LNetError: 90916:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1695.950023] LNet: Removed LNI 192.168.203.22@tcp [ 1695.996643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1696.092622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1696.142841] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1702.801567] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 21:25:59 (1781227559) [ 1702.901343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1702.939942] LNet: 91613:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1702.947527] LNetError: 91613:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1702.987144] LNet: Removed LNI 192.168.203.22@tcp [ 1703.009222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1703.073598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1703.125729] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1709.535727] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 21:26:06 (1781227566) [ 1710.831645] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1712.260552] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 21:26:09 (1781227569) [ 1712.808867] LNet: 92653:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1712.814181] LNetError: 92653:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1712.836978] LNet: Removed LNI 192.168.203.22@tcp [ 1713.399885] Key type .llcrypt unregistered [ 1713.407334] Key type ._llcrypt unregistered [ 1719.886490] Key type ._llcrypt registered [ 1719.890153] Key type .llcrypt registered [ 1720.115767] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1720.128466] alg: No test for adler32 (adler32-zlib) [ 1721.067126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1721.118218] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1728.078755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.122@tcp [ 1732.369590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.203.122@tcp [ 1732.737718] LNet: 93070:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1732.748337] LNetError: 93070:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1732.773319] LNet: Removed LNI 192.168.203.22@tcp [ 1733.107461] Key type .llcrypt unregistered [ 1733.113682] Key type ._llcrypt unregistered [ 1743.863947] Key type ._llcrypt registered [ 1743.866630] Key type .llcrypt registered [ 1745.187668] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1746.671295] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1748.586693] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 21:26:45 (1781227605) [ 1750.160607] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1752.174373] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 21:26:48 (1781227608) [ 1754.286622] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg322-server [ 1756.314856] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 21:26:52 (1781227612) [ 1758.477991] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg322-server [ 1760.418620] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 21:26:57 (1781227617) [ 1761.787872] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1761.840326] alg: No test for adler32 (adler32-zlib) [ 1763.063348] Lustre: Lustre: Build Version: 2.17.53_68_ga4ebd63 [ 1763.382788] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1765.085165] Key type lgssc registered [ 1766.608599] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1787.411734] Key type lgssc unregistered [ 1787.775646] LNet: 96258:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1787.780805] LNetError: 96258:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1787.815152] LNet: Removed LNI 192.168.203.22@tcp [ 1788.649503] Key type .llcrypt unregistered [ 1788.655803] Key type ._llcrypt unregistered [ 1804.427705] Key type ._llcrypt registered [ 1804.430106] Key type .llcrypt registered [ 1805.731890] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 21:27:42 (1781227662) [ 1806.250458] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1806.266142] alg: No test for adler32 (adler32-zlib) [ 1807.305960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1807.401287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1807.462872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1807.522700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1807.571231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1814.211312] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 21:27:51 (1781227671) [ 1814.341235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1814.393369] LNet: 97670:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1814.460248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1814.537205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1814.560100] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1815.326804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1815.352378] LNet: 97817:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1815.362327] LNetError: 97817:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1815.397276] LNet: Removed LNI 192.168.203.22@tcp [ 1815.434151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1815.529837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1815.562145] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1816.264828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1816.283890] LNetError: 97845:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1816.294910] LNet: Removed LNI 192.168.203.22@tcp [ 1816.320532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1816.375176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1816.425222] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1823.645268] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 21:28:00 (1781227680) [ 1824.019626] LNet: 98437:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1824.025239] LNet: 98437:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1824.031129] LNetError: 98437:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1824.059421] LNet: Removed LNI 192.168.203.22@tcp [ 1824.807162] Key type .llcrypt unregistered [ 1824.809949] Key type ._llcrypt unregistered [ 1830.986893] Key type ._llcrypt registered [ 1830.993280] Key type .llcrypt registered [ 1831.214575] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1831.232800] alg: No test for adler32 (adler32-zlib) [ 1832.164670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1832.250896] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1832.259056] LNet: *** cfs_fail_loc=e003, val=100*** [ 1833.043322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.22@tcp [ 1833.078307] LNet: *** cfs_fail_loc=e003, val=100*** [ 1833.082077] LNet: Skipped 2 previous similar messages [ 1833.087517] Ping reply received with mapping: [ 1833.090580] found metadata reply NI[192.168.203.22@tcp] [ 1833.098811] 16 cb a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1833.102421] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1833.105030] 5a 5a 5a 5a 5a 5a 5a 5a [ 1833.236791] LNet: 98835:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1833.242747] LNetError: 98835:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1833.259254] LNet: Removed LNI 192.168.203.22@tcp [ 1833.859400] Key type .llcrypt unregistered [ 1833.861466] Key type ._llcrypt unregistered [ 1845.200868] Key type ._llcrypt registered [ 1845.203843] Key type .llcrypt registered [ 1846.520615] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 21:28:23 (1781227703) [ 1847.112259] Key type .llcrypt unregistered [ 1847.113899] Key type ._llcrypt unregistered [ 1854.267398] Key type ._llcrypt registered [ 1854.276239] Key type .llcrypt registered [ 1854.616823] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1854.643942] alg: No test for adler32 (adler32-zlib) [ 1855.841248] LNetError: 99881:0:(api-ni.c:2729:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1856.252676] Key type .llcrypt unregistered [ 1856.258326] Key type ._llcrypt unregistered [ 1861.762249] Key type ._llcrypt registered [ 1861.764541] Key type .llcrypt registered [ 1863.131675] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 21:28:39 (1781227719) [ 1863.822610] Key type .llcrypt unregistered [ 1863.828651] Key type ._llcrypt unregistered [ 1869.572494] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1869.578081] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1872.095141] Key type ._llcrypt registered [ 1872.097164] Key type .llcrypt registered [ 1872.227732] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1872.237898] alg: No test for adler32 (adler32-zlib) [ 1873.187085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1873.269684] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1882.896694] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 21:28:59 (1781227739) [ 1882.982722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1883.014147] LNet: 101657:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1883.035583] LNetError: 101657:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1883.088046] LNet: Removed LNI 10.1.2.3@tcp [ 1883.145585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1884.104561] Key type .llcrypt unregistered [ 1884.111073] Key type ._llcrypt unregistered [ 1890.130247] Key type ._llcrypt registered [ 1890.133753] Key type .llcrypt registered [ 1890.420663] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1890.448100] alg: No test for adler32 (adler32-zlib) [ 1891.413201] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1891.500778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.22@tcp [ 1891.715697] LNet: 101950:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1891.724299] LNetError: 101950:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1892.777561] LNet: Removed LNI 192.168.203.22@tcp [ 1893.403584] Key type .llcrypt unregistered [ 1893.410928] Key type ._llcrypt unregistered [ 1906.880769] Key type ._llcrypt registered [ 1906.902277] Key type .llcrypt registered [ 1908.630640] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 21:29:24 (1781227764) [ 1909.162658] Key type .llcrypt unregistered [ 1909.166689] Key type ._llcrypt unregistered [ 1916.778348] Key type ._llcrypt registered [ 1916.783493] Key type .llcrypt registered [ 1917.095734] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1917.133563] alg: No test for adler32 (adler32-zlib) [ 1924.543992] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 21:29:41 (1781227781) [ 1924.611611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1924.644842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1924.726236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1924.763965] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1925.568731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1925.611458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1925.651199] LNet: Removed LNI 192.168.203.22@tcp [ 1926.005510] LNet: 103694:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1926.455536] Key type .llcrypt unregistered [ 1926.458828] Key type ._llcrypt unregistered [ 1937.721975] Key type ._llcrypt registered [ 1937.728446] Key type .llcrypt registered [ 1939.430330] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 21:29:55 (1781227795) [ 1939.945087] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1939.970523] alg: No test for adler32 (adler32-zlib) [ 1940.979713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1941.035036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1941.067416] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1942.015443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.22@tcp [ 1942.050415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.203.22@tcp [ 1942.067650] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1942.071370] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1942.074144] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1942, 0, 0 [ 1942.077298] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1942.081360] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1942.086121] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1942.091889] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1942, 0, 0 [ 1942.099064] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1942.102197] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1942.108251] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1942.112039] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1942, 0, 1992 [ 1942.115012] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1942.125560] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1942.130028] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1942.135836] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1942, 0, 1992 [ 1942.142366] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1948.320977] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 21:30:04 (1781227804) [ 1948.845200] LNet: 105233:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1948.854232] LNetError: 105233:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1948.867622] LNet: Removed LNI 192.168.203.22@tcp [ 1949.551384] Key type .llcrypt unregistered [ 1949.554577] Key type ._llcrypt unregistered [ 1956.162106] Key type ._llcrypt registered [ 1956.170442] Key type .llcrypt registered [ 1956.430606] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1956.444247] alg: No test for adler32 (adler32-zlib) [ 1957.405902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1957.440496] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1964.931259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.122@tcp [ 1974.110792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1974.183242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1974.212901] LNet: Added LNI 192.168.203.22@tcp1 [8/256/0/180] [ 1975.188565] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1975.395280] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1975.401854] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1975.405912] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1977.799493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1977.829291] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1988.968482] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 21:30:45 (1781227845) [ 1989.141579] LNet: 106736:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1989.160458] LNetError: 106736:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1989.176733] LNet: Removed LNI 192.168.203.22@tcp [ 1990.240282] LNet: Removed LNI 192.168.203.22@tcp1 [ 1990.247331] LNetError: 106736:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1990.259013] LNetError: 106736:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1990.289854] LNet: Removed LNI 10.1.2.3@tcp99 [ 1991.073562] Key type .llcrypt unregistered [ 1991.076680] Key type ._llcrypt unregistered [ 1998.343916] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1998.348659] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2001.041718] Key type ._llcrypt registered [ 2001.045964] Key type .llcrypt registered [ 2001.293639] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2001.309602] alg: No test for adler32 (adler32-zlib) [ 2002.253242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2002.313601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2002.331322] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 2003.063830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2003.081795] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2005.243992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2005.298159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2005.354363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2005.373695] LNetError: 107485:0:(peer.c:2175:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2005.415762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2005.454468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.22@tcp [ 2006.073738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2006.201343] LNet: 107505:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2006.225346] LNetError: 107505:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2007.264289] LNet: Removed LNI 192.168.203.22@tcp [ 2007.267082] LNetError: 107505:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2007.302316] LNet: Removed LNI 10.1.2.3@tcp [ 2008.083390] Key type .llcrypt unregistered [ 2008.085586] Key type ._llcrypt unregistered [ 2021.723599] Key type ._llcrypt registered [ 2021.725716] Key type .llcrypt registered [ 2023.061836] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 21:31:19 (1781227879) [ 2023.531693] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2023.555136] alg: No test for adler32 (adler32-zlib) [ 2024.504170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2024.583396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2024.612098] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 2025.357265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.22@tcp [ 2025.426709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg322-client@tcp [ 2031.680901] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 21:31:28 (1781227888) [ 2031.754856] LNet: 109031:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2031.771900] LNetError: 109031:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2031.789778] LNet: Removed LNI 192.168.203.22@tcp [ 2032.519385] Key type .llcrypt unregistered [ 2032.526917] Key type ._llcrypt unregistered [ 2038.693625] Key type ._llcrypt registered [ 2038.698668] Key type .llcrypt registered [ 2038.904353] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2038.929645] alg: No test for adler32 (adler32-zlib) [ 2039.833154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2039.855321] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 2039.883825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2047.228421] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 21:31:43 (1781227903) [ 2049.175251] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2050.824907] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 21:31:47 (1781227907) [ 2052.870231] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2054.901185] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 21:31:51 (1781227911) [ 2055.152598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2055.174464] LNet: 110459:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2055.180739] LNetError: 110459:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2055.194730] LNet: Removed LNI 192.168.203.22@tcp [ 2055.209386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2055.261383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2055.277866] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 2056.121160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2056.161640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2063.069314] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 21:31:59 (1781227919) [ 2063.230153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2063.275790] LNet: 111163:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2063.291117] LNetError: 111163:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2063.345974] LNet: Removed LNI 192.168.203.22@tcp [ 2063.427139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2063.509783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2063.601638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2064.516623] Key type .llcrypt unregistered [ 2064.521324] Key type ._llcrypt unregistered [ 2078.316369] Key type ._llcrypt registered [ 2078.318704] Key type .llcrypt registered [ 2079.712684] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 21:32:16 (1781227936) [ 2080.119411] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2080.133594] alg: No test for adler32 (adler32-zlib) [ 2081.101826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2081.186318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2081.256280] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 2082.136608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2082.228983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.22@tcp [ 2089.267148] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 21:32:25 (1781227945) [ 2089.382939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2089.422677] LNet: 112718:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2089.433668] LNetError: 112718:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2089.470568] LNet: Removed LNI 192.168.203.22@tcp [ 2089.522727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2089.594171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2089.617548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2095.933353] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 21:32:32 (1781227952) [ 2096.140716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2096.166275] LNet: 113305:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2096.186201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2096.264124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2096.696854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2096.753713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2096.846803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2097.459827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2097.500207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2097.587796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2097.606827] LNet: 113338:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2097.627833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2097.691523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2098.055209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2098.089245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2098.155864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2098.684392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2098.729466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2105.243662] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 21:32:41 (1781227961) [ 2105.362375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2105.405779] LNet: 113935:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2105.446884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2105.528782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2105.569632] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 2106.295262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2106.319318] LNet: Added LNI 192.168.203.22@tcp2 [8/256/0/180] [ 2107.390499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2107.466155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.22@tcp [ 2107.739422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2107.781795] LNetError: 114218:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2107.794400] LNet: Removed LNI 192.168.203.22@tcp [ 2107.834876] LNet: Removed LNI 192.168.203.22@tcp2 [ 2107.862126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2107.930467] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 2107.938071] LNet: Added LNI 192.168.203.22@tcp2 [8/256/0/180] [ 2107.944841] LNet: 114224:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2108.594558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.22@tcp [ 2108.848686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2108.905481] LNetError: 114262:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2108.910891] LNetError: 114262:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2108.923107] LNet: Removed LNI 192.168.203.22@tcp [ 2108.942739] LNet: Removed LNI 192.168.203.22@tcp2 [ 2108.973995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2109.063970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2109.101776] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 2110.057766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2110.083774] LNet: Added LNI 192.168.203.22@tcp2 [8/256/0/180] [ 2111.358904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.22@tcp [ 2111.403520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2111.456613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.22@tcp [ 2111.746887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2111.793872] LNet: 114546:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2111.800456] LNet: 114546:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2111.808926] LNetError: 114546:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2111.822317] LNetError: 114546:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2111.833902] LNet: Removed LNI 192.168.203.22@tcp [ 2112.881686] LNet: Removed LNI 192.168.203.22@tcp2 [ 2112.943764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2113.053612] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 2113.061403] LNet: Added LNI 192.168.203.22@tcp2 [8/256/0/180] [ 2113.066739] LNet: 114552:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2113.768169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.22@tcp [ 2114.049215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2114.073774] LNet: Removed LNI 192.168.203.22@tcp [ 2115.111088] LNet: Removed LNI 192.168.203.22@tcp2 [ 2115.156208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2115.258546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2115.296219] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 2116.163307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2116.193088] LNet: Added LNI 192.168.203.22@tcp2 [8/256/0/180] [ 2117.195376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2117.260547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.22@tcp [ 2117.533549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2117.552370] LNetError: 114869:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2117.558342] LNetError: 114869:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2117.564713] LNet: Removed LNI 192.168.203.22@tcp [ 2117.574573] LNet: Removed LNI 192.168.203.22@tcp2 [ 2117.610683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2117.684633] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 2117.692287] LNet: Added LNI 192.168.203.22@tcp2 [8/256/0/180] [ 2117.695502] LNet: 114875:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2118.383463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.22@tcp [ 2118.648770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2119.712268] LNet: Removed LNI 192.168.203.22@tcp [ 2119.727040] LNet: Removed LNI 192.168.203.22@tcp2 [ 2119.809780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2119.898820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2119.945409] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 2120.924426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2120.954106] LNet: Added LNI 192.168.203.22@tcp2 [8/256/0/180] [ 2122.114425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.22@tcp [ 2122.180434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2122.271153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.22@tcp [ 2122.612607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2122.640351] LNet: 115197:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2122.647897] LNet: 115197:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2123.684038] LNet: Removed LNI 192.168.203.22@tcp [ 2124.717524] LNet: Removed LNI 192.168.203.22@tcp2 [ 2124.739235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2124.823819] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 2124.831853] LNet: Added LNI 192.168.203.22@tcp2 [8/256/0/180] [ 2124.842410] LNet: 115203:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2125.407407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.22@tcp [ 2125.668952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2125.691121] LNetError: 115240:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2125.694570] LNetError: 115240:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2125.698900] LNet: Removed LNI 192.168.203.22@tcp [ 2126.759136] LNet: Removed LNI 192.168.203.22@tcp2 [ 2126.771272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2126.806295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2126.818748] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 2127.503985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2127.529799] LNet: Added LNI 192.168.203.22@tcp2 [8/256/0/180] [ 2128.472663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2128.501526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2128.543803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.22@tcp [ 2128.879290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2129.952156] LNet: Removed LNI 192.168.203.22@tcp [ 2129.972679] LNet: Removed LNI 192.168.203.22@tcp2 [ 2130.062995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2130.116451] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 2130.122647] LNet: Added LNI 192.168.203.22@tcp2 [8/256/0/180] [ 2130.127370] LNet: 115543:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2130.805979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.22@tcp [ 2131.375217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2132.449834] LNet: Removed LNI 192.168.203.22@tcp [ 2132.480809] LNet: Removed LNI 192.168.203.22@tcp2 [ 2132.549373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2132.657431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2132.719594] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 2133.682571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2133.734589] LNet: Added LNI 192.168.203.22@tcp2 [8/256/0/180] [ 2134.697869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.22@tcp [ 2134.734336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2134.789383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2134.895881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.22@tcp [ 2135.248352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2136.288142] LNet: Removed LNI 192.168.203.22@tcp [ 2137.323616] LNet: Removed LNI 192.168.203.22@tcp2 [ 2137.368251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2137.435349] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 2137.453938] LNet: Added LNI 192.168.203.22@tcp2 [8/256/0/180] [ 2137.816373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.22@tcp [ 2146.185200] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2148.451628] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 21:33:24 (1781228004) [ 2148.686419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2148.719262] LNet: 116659:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2148.732370] LNet: 116659:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2148.739592] LNetError: 116659:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2148.744613] LNetError: 116659:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2148.750869] LNet: Removed LNI 192.168.203.22@tcp [ 2149.810302] LNet: Removed LNI 192.168.203.22@tcp2 [ 2149.842984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2150.102616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2150.138577] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 2150.938621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2150.982947] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2152.008683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2152.025546] LNet: Removed LNI 192.168.203.22@tcp [ 2152.036822] LNet: Removed LNI 10.1.2.3@tcp [ 2152.054816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2152.087344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2152.100539] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 2152.103760] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2152.410156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2152.426861] LNet: Removed LNI 192.168.203.22@tcp [ 2152.441492] LNet: Removed LNI 10.1.2.3@tcp [ 2158.814118] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 21:33:35 (1781228015) [ 2159.181742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2159.228155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2159.314958] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2159.409382] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2159.414839] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2159.418151] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2161.746994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2161.775904] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 2162.654578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2162.668411] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2164.794673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.22@tcp [ 2164.850674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2164.890115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.203.22@tcp -m PUT -e local_timeout -r 1 [ 2164.983262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2165.056104] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2165.060663] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2165.304747] LNetError: 117603:0:(lib-move.c:4710:lnet_parse()) 192.168.203.22@tcp, src 192.168.203.22@tcp: Dropping PUT (error -108 looking up sender) [ 2167.328249] LNet: Removed LNI 192.168.203.22@tcp [ 2167.342619] LNet: Removed LNI 10.1.2.3@tcp [ 2167.788043] Key type .llcrypt unregistered [ 2167.793150] Key type ._llcrypt unregistered [ 2178.927179] Key type ._llcrypt registered [ 2178.928950] Key type .llcrypt registered [ 2180.520947] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 21:33:56 (1781228036) [ 2181.266779] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2181.284732] alg: No test for adler32 (adler32-zlib) [ 2182.228612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2182.741824] LNet: 118916:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2183.470351] Key type .llcrypt unregistered [ 2183.472984] Key type ._llcrypt unregistered [ 2190.447671] Key type ._llcrypt registered [ 2190.449366] Key type .llcrypt registered [ 2190.737901] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2190.757489] alg: No test for adler32 (adler32-zlib) [ 2191.734189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2191.762989] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 2199.294712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.122@tcp [ 2199.527336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2200.090574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.122@tcp [ 2200.333068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2200.365211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2200.394821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2200.454125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2200.508351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2200.559182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2200.617963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2200.655868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2200.682470] LNet: 119366:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2200.714241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2200.775312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2200.813925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2200.858193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2200.919724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2200.973805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2201.636662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2201.859713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2201.895809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2201.941329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2201.974169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2202.002120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2202.019980] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.122@tcp [ 2202.049332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2202.098776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2202.155328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2202.173668] LNet: 119484:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2202.207385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2202.258750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2202.302225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2202.342252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2202.384762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2202.412970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2203.073234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.122@tcp [ 2203.379666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2203.408608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2203.458430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2203.489881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2203.537903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2203.574294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2203.621938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2203.653768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2203.672593] LNet: 119601:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2203.687866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2203.712782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2203.742971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2203.787404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2203.834358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2203.871532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2204.489322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2204.721626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2204.771902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2204.806496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2204.836047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2204.868538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2204.899888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2204.929153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2204.951581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2204.990946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2205.032195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2205.075628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2205.109957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2205.138434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2205.191271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2205.506942] LNet: 119764:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2205.511784] LNetError: 119764:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2206.568910] LNet: Removed LNI 192.168.203.22@tcp [ 2206.983959] Key type .llcrypt unregistered [ 2206.988721] Key type ._llcrypt unregistered [ 2223.572965] Key type ._llcrypt registered [ 2223.580444] Key type .llcrypt registered [ 2224.893971] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 21:34:41 (1781228081) [ 2225.639871] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2225.661609] alg: No test for adler32 (adler32-zlib) [ 2226.588658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2227.010041] LNet: 120653:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2227.704351] Key type .llcrypt unregistered [ 2227.706717] Key type ._llcrypt unregistered [ 2234.103700] Key type ._llcrypt registered [ 2234.106508] Key type .llcrypt registered [ 2234.319516] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2234.335990] alg: No test for adler32 (adler32-zlib) [ 2235.223625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2235.254688] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 2242.834976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.122@tcp [ 2252.474590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2252.512565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2252.545893] LNet: Added LNI 192.168.203.22@tcp1 [8/256/0/180] [ 2253.667988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2254.149734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.122@tcp [ 2254.337744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2254.365717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2254.394227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2254.420854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2254.451113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2254.475655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2254.517613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2254.558507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2254.580580] LNet: 121306:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2254.614652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2254.650687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2254.693718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2254.733309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2254.753636] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.122@tcp [ 2254.763990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2254.808867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2255.434544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2255.617474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2255.645960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2255.674220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2255.698586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2255.734948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2255.773316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2255.821575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2255.874244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2255.898106] LNet: 121423:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2255.913618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2255.962071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2256.007652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2256.046666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2256.099838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2256.130563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2256.715566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.122@tcp [ 2256.912538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2256.937633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2256.969931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2256.999984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2257.031606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2257.061990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2257.089716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2257.117561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2257.133530] LNet: 121540:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2257.163713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2257.189253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2257.226414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2257.258250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2257.289035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2257.337248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2258.001195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2258.300328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2258.337765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2258.388094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2258.422355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2258.492759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2258.530989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2258.573323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2258.610236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2258.648313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2258.696486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2258.768889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2258.833622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2258.867259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2258.926642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2263.982593] LNet: 121738:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2263.990777] LNetError: 121738:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2263.998860] LNet: Removed LNI 192.168.203.22@tcp [ 2264.011177] LNet: Removed LNI 192.168.203.22@tcp1 [ 2264.446147] Key type .llcrypt unregistered [ 2264.452933] Key type ._llcrypt unregistered [ 2281.365669] Key type ._llcrypt registered [ 2281.370485] Key type .llcrypt registered [ 2282.666937] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 21:35:39 (1781228139) [ 2289.878636] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 21:35:46 (1781228146) [ 2296.572649] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 21:35:53 (1781228153) [ 2303.543520] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 21:36:00 (1781228160) [ 2310.338118] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 21:36:07 (1781228167) [ 2317.800397] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 21:36:14 (1781228174) [ 2324.754581] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 21:36:21 (1781228181) [ 2332.407353] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 21:36:29 (1781228189) [ 2339.518677] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 21:36:35 (1781228195) [ 2348.467846] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 21:36:45 (1781228205) [ 2356.725350] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 21:36:53 (1781228213) [ 2363.968442] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 21:37:00 (1781228220) [ 2370.601731] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 21:37:07 (1781228227) [ 2377.883819] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 21:37:14 (1781228234) [ 2385.322636] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 21:37:22 (1781228242) [ 2392.468520] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 21:37:29 (1781228249) [ 2399.231553] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 21:37:36 (1781228256) [ 2407.030226] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 21:37:43 (1781228263) [ 2407.369837] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2407.380472] alg: No test for adler32 (adler32-zlib) [ 2408.335408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2408.512305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2408.542260] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 2409.772544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2409.826513] LNet: 132447:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2409.851558] LNetError: 132447:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2409.884396] LNet: Removed LNI 192.168.203.22@tcp [ 2409.914945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2410.010732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2410.092648] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 2417.522452] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 21:37:54 (1781228274) [ 2425.232204] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 21:38:02 (1781228282) [ 2433.235759] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 21:38:09 (1781228289) [ 2439.940788] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 21:38:16 (1781228296) [ 2446.297436] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 21:38:23 (1781228303) [ 2454.038417] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 21:38:30 (1781228310) [ 2460.569961] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 21:38:37 (1781228317) [ 2467.698332] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 21:38:44 (1781228324) [ 2475.356590] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 21:38:51 (1781228331) [ 2485.247711] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 21:39:01 (1781228341) [ 2493.542664] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 21:39:10 (1781228350) [ 2501.568079] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 21:39:17 (1781228357) [ 2509.288945] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2246 sec ========== 21:39:25 (1781228365) [ 2509.582820] LNet: 139914:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2509.590948] LNetError: 139914:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2509.606228] LNet: Removed LNI 192.168.203.22@tcp [ 2510.301618] Key type .llcrypt unregistered [ 2510.306649] Key type ._llcrypt unregistered [ 2523.328561] Key type ._llcrypt registered [ 2523.336681] Key type .llcrypt registered [ 2523.608978] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2523.623233] alg: No test for adler32 (adler32-zlib) [ 2524.792441] Lustre: Lustre: Build Version: 2.17.53_68_ga4ebd63 [ 2525.104128] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 2526.824315] Key type lgssc registered [ 2528.837278] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2592.798655] Lustre: Mounted lustre-client [ 2597.741630] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2605.825348] Lustre: Unmounted lustre-client [ 2623.749332] Key type lgssc unregistered [ 2623.957623] LNet: 142840:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2623.968870] LNetError: 142840:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2623.998134] LNet: Removed LNI 192.168.203.22@tcp [ 2624.796768] Key type .llcrypt unregistered [ 2624.803299] Key type ._llcrypt unregistered