[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 463232143 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.003370] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.008336] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009052] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010011] pid_max: default: 32768 minimum: 301 [ 0.011130] LSM: Security Framework initializing [ 0.012050] Yama: becoming mindful. [ 0.013050] SELinux: Initializing. [ 0.014070] *** VALIDATE selinux *** [ 0.021750] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026498] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028010] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030008] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031112] *** VALIDATE tmpfs *** [ 0.033234] *** VALIDATE proc *** [ 0.034242] *** VALIDATE cgroup *** [ 0.035011] *** VALIDATE cgroup2 *** [ 0.036268] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037162] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039028] Spectre V2 : User space: Vulnerable [ 0.040009] Speculative Store Bypass: Vulnerable [ 0.043147] debug: unmapping init [mem 0xffffffff8fa59000-0xffffffff8fa60fff] [ 0.046197] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047714] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048029] ... version: 2 [ 0.049014] ... bit width: 48 [ 0.050016] ... generic registers: 4 [ 0.051014] ... value mask: 0000ffffffffffff [ 0.052016] ... max period: 00007fffffffffff [ 0.053015] ... fixed-purpose events: 3 [ 0.054014] ... event mask: 000000070000000f [ 0.056224] rcu: Hierarchical SRCU implementation. [ 0.058453] smp: Bringing up secondary CPUs ... [ 0.059643] x86: Booting SMP configuration: [ 0.060027] .... node #0, CPUs: #1 #2 #3 [ 0.064140] smp: Brought up 1 node, 4 CPUs [ 0.066019] smpboot: Max logical packages: 1 [ 0.067021] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.246025] node 0 deferred pages initialised in 177ms [ 0.250174] devtmpfs: initialized [ 0.251222] x86/mm: Memory block size: 128MB [ 0.254625] gcov: version magic: 0x41383552 [ 0.256299] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.257090] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.258261] pinctrl core: initialized pinctrl subsystem [ 0.259206] [ 0.259738] ************************************************************* [ 0.260020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.261014] ** ** [ 0.262027] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.263018] ** ** [ 0.264015] ** This means that this kernel is built to expose internal ** [ 0.265017] ** IOMMU data structures, which may compromise security on ** [ 0.266017] ** your system. ** [ 0.267022] ** ** [ 0.268021] ** If you see this message and you are not debugging the ** [ 0.269021] ** kernel, report this immediately to your vendor! ** [ 0.270013] ** ** [ 0.271015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.272016] ************************************************************* [ 0.273642] NET: Registered protocol family 16 [ 0.274415] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.275062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.276062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.277450] cpuidle: using governor menu [ 0.279839] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.281613] PCI: Using configuration type 1 for base access [ 0.284118] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.294112] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.295023] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.297109] cryptd: max_cpu_qlen set to 1000 [ 0.301047] ACPI: Added _OSI(Module Device) [ 0.302021] ACPI: Added _OSI(Processor Device) [ 0.305030] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.307019] ACPI: Added _OSI(Processor Aggregator Device) [ 0.312742] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.319546] ACPI: Interpreter enabled [ 0.321089] ACPI: PM: (supports S0 S3 S4 S5) [ 0.323018] ACPI: Using IOAPIC for interrupt routing [ 0.325136] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.330441] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.341000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.344055] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.347026] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.351112] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.356221] acpiphp: Slot [2] registered [ 0.357158] acpiphp: Slot [5] registered [ 0.359141] acpiphp: Slot [6] registered [ 0.362179] acpiphp: Slot [3] registered [ 0.363112] acpiphp: Slot [4] registered [ 0.364083] acpiphp: Slot [7] registered [ 0.365088] acpiphp: Slot [8] registered [ 0.367122] acpiphp: Slot [9] registered [ 0.368128] acpiphp: Slot [10] registered [ 0.370146] acpiphp: Slot [11] registered [ 0.371167] acpiphp: Slot [12] registered [ 0.373144] acpiphp: Slot [13] registered [ 0.375142] acpiphp: Slot [14] registered [ 0.377135] acpiphp: Slot [15] registered [ 0.378135] acpiphp: Slot [16] registered [ 0.380146] acpiphp: Slot [17] registered [ 0.382139] acpiphp: Slot [18] registered [ 0.383126] acpiphp: Slot [19] registered [ 0.385154] acpiphp: Slot [20] registered [ 0.386140] acpiphp: Slot [21] registered [ 0.388109] acpiphp: Slot [22] registered [ 0.389098] acpiphp: Slot [23] registered [ 0.391167] acpiphp: Slot [24] registered [ 0.392156] acpiphp: Slot [25] registered [ 0.394139] acpiphp: Slot [26] registered [ 0.396172] acpiphp: Slot [27] registered [ 0.397110] acpiphp: Slot [28] registered [ 0.399137] acpiphp: Slot [29] registered [ 0.400089] acpiphp: Slot [30] registered [ 0.402169] acpiphp: Slot [31] registered [ 0.405100] PCI host bridge to bus 0000:00 [ 0.407036] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.410036] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.413045] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.416049] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.419036] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.421033] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.423222] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.427424] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.431416] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.440022] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.444064] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.447026] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.450031] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.453032] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.456794] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.459892] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.463065] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.466729] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.471020] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.482020] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.486020] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.492601] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.497997] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.505016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.522014] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.533894] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.543023] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.549018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.569021] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.582152] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.585464] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.588379] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.591461] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.594285] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.600068] iommu: Default domain type: Passthrough [ 0.602567] SCSI subsystem initialized [ 0.604186] ACPI: bus type USB registered [ 0.605161] usbcore: registered new interface driver usbfs [ 0.608104] usbcore: registered new interface driver hub [ 0.609112] usbcore: registered new device driver usb [ 0.612248] pps_core: LinuxPPS API ver. 1 registered [ 0.614016] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.617091] PTP clock support registered [ 0.619161] EDAC MC: Ver: 3.0.0 [ 0.621465] PCI: Using ACPI for IRQ routing [ 0.623902] NetLabel: Initializing [ 0.625017] NetLabel: domain hash size = 128 [ 0.627012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.629093] NetLabel: unlabeled traffic allowed by default [ 0.631161] vgaarb: loaded [ 0.633363] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.635022] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.641584] clocksource: Switched to clocksource kvm-clock [ 0.753217] VFS: Disk quotas dquot_6.6.0 [ 0.754865] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.757701] *** VALIDATE ramfs *** [ 0.759056] *** VALIDATE hugetlbfs *** [ 0.760867] pnp: PnP ACPI init [ 0.763341] pnp: PnP ACPI: found 6 devices [ 0.782607] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.786293] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.788864] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.791339] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.794201] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.796877] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.800232] NET: Registered protocol family 2 [ 0.803164] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.808607] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.812816] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.818637] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.822574] TCP: Hash tables configured (established 65536 bind 65536) [ 0.825942] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.829581] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.832879] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.836343] NET: Registered protocol family 1 [ 0.839306] RPC: Registered named UNIX socket transport module. [ 0.841932] RPC: Registered udp transport module. [ 0.843856] RPC: Registered tcp transport module. [ 0.845850] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.848772] NET: Registered protocol family 44 [ 0.850622] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.853056] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.855505] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.858120] PCI: CLS 0 bytes, default 64 [ 0.859964] Unpacking initramfs... [ 2.302537] debug: unmapping init [mem 0xffff9e4dfcc64000-0xffff9e4dfffcffff] [ 2.309058] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.311255] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.315384] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.831051] Initialise system trusted keyrings [ 2.833305] Key type blacklist registered [ 2.835568] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.844361] zbud: loaded [ 2.847593] *** VALIDATE nfs *** [ 2.849089] *** VALIDATE nfs4 *** [ 2.851758] pstore: using deflate compression [ 2.855072] Platform Keyring initialized [ 2.960366] NET: Registered protocol family 38 [ 2.962261] Key type asymmetric registered [ 2.965058] Asymmetric key parser 'x509' registered [ 2.967163] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.970895] io scheduler mq-deadline registered [ 2.972729] io scheduler kyber registered [ 2.974087] io scheduler bfq registered [ 2.976700] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.980324] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.983740] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.987327] ACPI: Power Button [PWRF] [ 2.993649] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.000551] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.011712] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.041694] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.070305] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.077360] Non-volatile memory driver v1.3 [ 3.079340] Linux agpgart interface v0.103 [ 3.111269] virtio_blk virtio1: [vda] 146152 512-byte logical blocks (74.8 MB/71.4 MiB) [ 3.114626] vda: detected capacity change from 0 to 74829824 [ 3.133111] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.136641] vdb: detected capacity change from 0 to 1073741824 [ 3.145115] libphy: Fixed MDIO Bus: probed [ 3.150887] usbcore: registered new interface driver usbserial_generic [ 3.153867] usbserial: USB Serial support registered for generic [ 3.156664] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.161786] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.163950] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.167222] mousedev: PS/2 mouse device common for all mice [ 3.169937] rtc_cmos 00:05: RTC can wake from S4 [ 3.172864] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.175217] rtc_cmos 00:05: registered as rtc0 [ 3.181087] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.181133] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.186920] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.188468] intel_pstate: CPU model not supported [ 3.196582] hid: raw HID events driver (C) Jiri Kosina [ 3.198700] usbcore: registered new interface driver usbhid [ 3.201141] usbhid: USB HID core driver [ 3.203119] drop_monitor: Initializing network drop monitor service [ 3.206188] Initializing XFRM netlink socket [ 3.208448] NET: Registered protocol family 10 [ 3.213173] Segment Routing with IPv6 [ 3.215157] NET: Registered protocol family 17 [ 3.217596] mpls_gso: MPLS GSO support [ 3.223661] RAS: Correctable Errors collector initialized. [ 3.225747] AVX version of gcm_enc/dec engaged. [ 3.227718] AES CTR mode by8 optimization enabled [ 3.308088] sched_clock: Marking stable (3308038113, 0)->(4253572214, -945534101) [ 3.311954] registered taskstats version 1 [ 3.314244] Loading compiled-in X.509 certificates [ 3.317422] zswap: loaded using pool lzo/zbud [ 3.343261] Key type big_key registered [ 3.356339] Key type encrypted registered [ 3.358317] ima: No TPM chip found, activating TPM-bypass! [ 3.361016] ima: Allocated hash algorithm: sha1 [ 3.363237] ima: No architecture policies found [ 3.365422] evm: Initialising EVM extended attributes: [ 3.367588] evm: security.selinux [ 3.368952] evm: security.ima [ 3.370063] evm: security.capability [ 3.371485] evm: HMAC attrs: 0x1 [ 3.373871] rtc_cmos 00:05: setting system clock to 2026-08-24 19:17:45 UTC (1787599065) [ 3.380415] debug: unmapping init [mem 0xffffffff90a03000-0xffffffff90bfffff] [ 3.384054] debug: unmapping init [mem 0xffffffff8f782000-0xffffffff8fa58fff] [ 3.391080] Write protecting the kernel read-only data: 28672k [ 3.395266] debug: unmapping init [mem 0xffffffff8de03000-0xffffffff8dffffff] [ 3.398544] debug: unmapping init [mem 0xffffffff8e714000-0xffffffff8e7fffff] [ 3.433874] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.444394] systemd[1]: Detected virtualization kvm. [ 3.446567] systemd[1]: Detected architecture x86-64. [ 3.448813] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.474227] systemd[1]: No hostname configured. [ 3.475780] systemd[1]: Set hostname to . [ 3.477750] random: systemd: uninitialized urandom read (16 bytes read) [ 3.480650] systemd[1]: Initializing machine ID from random generator. [ 3.534795] random: ln: uninitialized urandom read (6 bytes read) [ 3.618104] random: systemd: uninitialized urandom read (16 bytes read) [ 3.621548] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.627523] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.632613] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. [ OK ] Reached target Sockets. Starting Journal Service... Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.300329] device-mapper: uevent: version 1.0.3 [ 4.302664] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.002856] random: fast init done Starting dracut initqueue hook... [ 5.079330] virtio_net virtio0 ens2: renamed from eth0 [ 5.134144] scsi host0: ata_piix [ 5.177451] scsi host1: ata_piix [ 5.180844] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.183511] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.998363] dracut-initqueue[580]: RTNETLINK answers: File exists [ 9.876877] random: crng init done [ 9.878954] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.332882] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.411455] printk: systemd: 24 output lines suppressed due to ratelimiting [ 11.653547] SELinux: Disabled at runtime. [ 11.712866] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.722927] systemd[1]: Detected virtualization kvm. [ 11.725973] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.176494] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.179306] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.183870] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.188351] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.191265] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.198747] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.203562] systemd[1]: Created slice User and Session Slice. [ OK ] Created slice User and Session Slice. [ OK ] Listening on udev Control Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Mounting Huge Pages File System... [ 12.235420] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Kernel Debug File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-sshd\x2dkeygen.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. Mounting POSIX Message Queue File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Apply Kernel Variables... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-getty.slice. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 12.658118] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.972434] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.036638] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.117249] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.129658] EDAC sbridge: Ver: 1.1.2 [ 16.628440] Key type dns_resolver registered [ 17.360449] NFS: Registering the id_resolver key type [ 17.365626] Key type id_resolver registered [ 17.371241] Key type id_legacy registered [* ] A start job is running for Configur…-only root support (5s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Load/Save Random Seed. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg342-client login: [ 38.455084] hrtimer: interrupt took 3076012 ns [ 89.333715] libcfs: loading out-of-tree module taints kernel. [ 89.508777] Key type ._llcrypt registered [ 89.513587] Key type .llcrypt registered [ 89.889674] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 89.908534] alg: No test for adler32 (adler32-zlib) [ 91.511562] Lustre: Lustre: Build Version: 2.17.57_80_g20f51fc [ 92.607433] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 94.440353] Key type lgssc registered [ 96.408489] Lustre: Echo OBD driver; http://www.lustre.org/ [ 227.125066] Lustre: Mounted lustre-client - version 2.17.57_80_g20f51fc [ 233.831833] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 250.472321] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing check_logdir /tmp/testlogs/ [ 252.900471] Lustre: lustre-OST0000-osc-ffff9e4e45016000: disconnect after 23s idle [ 254.929357] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing yml_node [ 259.605757] Lustre: DEBUG MARKER: Client: 2.17.57.80 [ 262.633975] Lustre: DEBUG MARKER: MDS: 2.17.57.80 [ 265.456990] Lustre: DEBUG MARKER: OSS: 2.17.57.80 [ 266.819177] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Mon Aug 24 15:22:07 EDT 2026 [ 283.531314] Lustre: Unmounted lustre-client [ 305.817138] Key type lgssc unregistered [ 306.098270] LNet: 8420:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 306.103932] LNetError: 8420:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 306.126331] LNet: Removed LNI 192.168.203.42@tcp [ 306.955150] Key type .llcrypt unregistered [ 306.957659] Key type ._llcrypt unregistered [ 333.535394] Key type ._llcrypt registered [ 333.539676] Key type .llcrypt registered [ 333.795800] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 333.809639] alg: No test for adler32 (adler32-zlib) [ 334.768344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 334.852204] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 334.877945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 338.966671] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 339.215372] LNet: 9310:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 339.226202] LNetError: 9310:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 339.244241] LNet: Removed LNI 192.168.203.42@tcp [ 339.746603] Key type .llcrypt unregistered [ 339.749893] Key type ._llcrypt unregistered [ 345.512369] Key type ._llcrypt registered [ 345.517711] Key type .llcrypt registered [ 346.541831] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 15:23:27 (1787599407) [ 347.352489] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 347.370755] alg: No test for adler32 (adler32-zlib) [ 348.348615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 348.472972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 348.504526] LNet: 9726:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 348.537391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 348.656679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 348.699133] LNet: 9733:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 356.337828] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 15:23:37 (1787599417) [ 356.581209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 356.619190] LNet: 10312:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 356.646624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 356.783463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 364.533228] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 15:23:45 (1787599425) [ 364.996646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 365.042248] LNet: 10901:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 365.078583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 365.234764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 372.659955] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 15:23:53 (1787599433) [ 372.883871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 372.915197] LNet: 11486:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 372.939552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 373.019918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 380.244425] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 15:24:00 (1787599440) [ 380.341050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 380.387583] LNet: 12064:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 380.417683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 380.561854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 381.235405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 388.043633] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 15:24:08 (1787599448) [ 388.153976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 388.226496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 388.363642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 388.760387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 389.177886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 395.019660] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 15:24:15 (1787599455) [ 395.162510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 395.214111] LNet: 13258:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 395.226128] LNet: 13258:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 395.258901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 395.336066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 402.041241] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 15:24:22 (1787599462) [ 402.338723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 402.384428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 402.458634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 402.506825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 403.025295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 403.062830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 403.565846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 403.614701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 404.023292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 404.048363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 404.420865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 404.459638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 404.933369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 405.006518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 405.465954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 405.492734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 405.776731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 405.812674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 406.234356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 406.271800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 406.779303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 406.810603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 407.210674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 407.259309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 407.746028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 407.782636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 408.092734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 408.123214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 414.280932] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 15:24:35 (1787599475) [ 414.468914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 414.484795] LNet: 14525:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 414.489411] LNet: 14525:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 414.505185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 414.579229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 414.635340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 420.946237] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 15:24:41 (1787599481) [ 421.244363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 421.297258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 421.431632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 421.493252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 428.244571] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 15:24:48 (1787599488) [ 428.357489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 428.399288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 428.474938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 428.514245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 435.090214] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 15:24:55 (1787599495) [ 435.205208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 435.249270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 435.307924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 435.344846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 441.717379] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 15:25:02 (1787599502) [ 441.838189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 441.888283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 441.984211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 442.024747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 449.174345] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 15:25:09 (1787599509) [ 449.266216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 449.294565] LNet: 17454:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 449.304113] LNet: 17454:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 449.335841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 449.429718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 449.482738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 455.967841] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 15:25:16 (1787599516) [ 456.214678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 456.244538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 456.314682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 456.665968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 463.431446] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 15:25:23 (1787599523) [ 463.590136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 463.670064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 464.031846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 464.286368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 471.420243] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 15:25:31 (1787599531) [ 471.613721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 471.687548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 471.828568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 472.047386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 478.406026] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 15:25:39 (1787599539) [ 478.716802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 478.779582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 479.219534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 479.442488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 485.695300] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 15:25:46 (1787599546) [ 485.916456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 485.988536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 486.102248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 486.142330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 492.185889] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 15:25:53 (1787599553) [ 492.478174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 492.552222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 492.628980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 492.697288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 499.060640] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 15:25:59 (1787599559) [ 499.339515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 499.370756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 499.425400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 505.450904] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 15:26:06 (1787599566) [ 505.884494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 505.941542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 506.041977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 513.323896] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 15:26:13 (1787599573) [ 513.422664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 513.460550] LNet: 22887:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 513.467284] LNet: 22887:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 513.497089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 513.635416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 514.198165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 520.481224] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 15:26:21 (1787599581) [ 520.607651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 520.634638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 520.695463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 520.756586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 526.960745] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 15:26:27 (1787599587) [ 527.133454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 527.176446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 527.258099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 527.298691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 532.790267] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 15:26:33 (1787599593) [ 532.932286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 533.005156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 533.113647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 533.163152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 538.718783] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 15:26:39 (1787599599) [ 538.824408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 538.870562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 538.945721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 538.974464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 545.987912] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 15:26:46 (1787599606) [ 546.196949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 546.229103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 546.281955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 546.359835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 546.384922] LNetError: 25841:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 546.426960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 546.459943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 552.315177] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 15:26:53 (1787599613) [ 552.429520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 552.473195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 552.586697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.RDGjgsOjox [ 552.680397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.RDGjgsOjox [ 552.779241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.RDGjgsOjox [ 559.183973] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 15:26:59 (1787599619) [ 559.497189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 559.533659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 559.600739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 559.653715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 559.724176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 566.610404] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 15:27:07 (1787599627) [ 566.815272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 566.893794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 566.985952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 567.013047] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 568.281734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.42@tcp [ 569.459509] Lustre: Lustre: Build Version: 2.17.57_80_g20f51fc [ 571.279246] Key type lgssc registered [ 572.280468] Lustre: Echo OBD driver; http://www.lustre.org/ [ 575.833748] LNetError: 27602:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.203.42@tcp99 has no available nets [ 578.292553] Key type lgssc unregistered [ 578.687410] LNetError: 28712:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 579.756408] LNet: Removed LNI 192.168.203.42@tcp [ 580.329571] Key type .llcrypt unregistered [ 580.334980] Key type ._llcrypt unregistered [ 587.738172] Key type ._llcrypt registered [ 587.742508] Key type .llcrypt registered [ 589.537586] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 15:27:29 (1787599649) [ 590.084512] Key type .llcrypt unregistered [ 590.088373] Key type ._llcrypt unregistered [ 597.581103] Key type ._llcrypt registered [ 597.589968] Key type .llcrypt registered [ 597.877171] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 597.905157] alg: No test for adler32 (adler32-zlib) [ 598.904972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 598.971721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 598.998377] LNet: 29634:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 599.009257] LNet: 29634:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 599.019956] LNet: 29634:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 599.040922] LNet: Message forwarding will be enabled in 2s [ 602.143125] LNet: Message forwarding enabled [ 602.600736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 602.636715] LNet: Message forwarding will stop in 2s to 6s [ 606.248875] LNet: Message forwarding disabled [ 607.182865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 607.196908] LNet: 29732:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 607.204601] LNet: 29732:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 607.210783] LNet: 29732:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 607.224035] LNet: Message forwarding will be enabled in 2s [ 607.431793] LNet: 29745:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 608.051657] Key type .llcrypt unregistered [ 608.053916] Key type ._llcrypt unregistered [ 620.792988] Key type ._llcrypt registered [ 620.799100] Key type .llcrypt registered [ 621.939311] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 15:28:02 (1787599682) [ 622.581214] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 622.596686] alg: No test for adler32 (adler32-zlib) [ 623.511705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 623.610939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 630.141979] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 15:28:10 (1787599690) [ 630.253627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 630.281181] LNet: 31144:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 630.309555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 630.405566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 630.470993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 630.505787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 630.568745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 630.593335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 630.628519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 630.677626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 630.748870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 630.808988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 630.839261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 630.882701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 630.931952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 630.973876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 631.034828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 631.114356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 631.176804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 631.232308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 631.274214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 631.344890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 631.385338] LNetError: 31194:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 637.035702] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 15:28:17 (1787599697) [ 637.252899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 637.270317] LNet: 31765:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 637.294405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 637.360118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 643.392807] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 15:28:24 (1787599704) [ 645.671782] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg342-server [ 647.794080] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 15:28:28 (1787599708) [ 647.971154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 648.038168] LNet: 32689:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 648.070427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 648.129602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 648.181895] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 649.266447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.203.[43-47]@tcp [ 649.301481] LNet: 32829:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 649.364444] LNetError: 32697:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.44@tcp is being used as a gateway but routing feature is not turned on [ 649.382897] LNetError: 32697:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.44@tcp has gone from up to down [ 656.788827] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 15:28:37 (1787599717) [ 658.312285] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg342-server [ 659.758922] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 15:28:40 (1787599720) [ 659.922421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 659.940148] LNet: 33765:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 659.945622] LNetError: 33765:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 660.977449] LNet: Removed LNI 192.168.203.42@tcp [ 661.014747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 661.065922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 661.087147] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 661.719522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.203.[43-48/2]@tcp [ 661.743678] LNet: 33883:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 661.753914] LNet: 33883:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 661.782582] LNetError: 33770:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.43@tcp is being used as a gateway but routing feature is not turned on [ 661.804948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.203.[43-48/2]@tcp [ 661.808332] LNetError: 33770:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 5 previous similar messages [ 661.808357] LNetError: 33770:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp103 through 192.168.203.43@tcp has gone from up to down [ 661.808362] LNetError: 33770:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 668.608910] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 15:28:49 (1787599729) [ 668.823834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 668.893945] LNet: 34476:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 668.906251] LNetError: 34476:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 668.929597] LNet: Removed LNI 192.168.203.42@tcp [ 668.954245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 669.063808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 669.131326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 669.183963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 669.277925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 669.330857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 669.378863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 669.440755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 669.524274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 669.552439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 669.603493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 669.666620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 669.691586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 669.749344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 669.858954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 669.912724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 670.041557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 670.133153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 670.169290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 670.214877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 670.327602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 670.356409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 670.444310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 670.556450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 670.599058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 670.668372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 670.788368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 670.844588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 670.930889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 670.999072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 671.037323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 671.118657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 671.208960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 671.253950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 671.328474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 677.543198] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 15:28:58 (1787599738) [ 679.355746] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg342-server [ 681.631894] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 15:29:02 (1787599742) [ 683.810712] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg342-server [ 685.281585] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 15:29:06 (1787599746) [ 685.631270] LNet: 35878:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 685.638859] LNet: 35878:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 686.270949] Key type .llcrypt unregistered [ 686.273752] Key type ._llcrypt unregistered [ 692.436831] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 692.583346] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 692.588643] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 692.598211] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 695.372672] Key type ._llcrypt registered [ 695.375314] Key type .llcrypt registered [ 695.541298] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 695.560522] alg: No test for adler32 (adler32-zlib) [ 696.460231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 696.495996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 696.513605] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 697.231269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 697.259248] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 699.510380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 699.538840] LNet: Removed LNI 10.1.2.3@tcp [ 699.719415] LNet: 36601:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 699.729924] LNetError: 36601:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 699.747595] LNet: Removed LNI 192.168.203.42@tcp [ 700.591757] Key type .llcrypt unregistered [ 700.594691] Key type ._llcrypt unregistered [ 715.241705] Key type ._llcrypt registered [ 715.251683] Key type .llcrypt registered [ 717.000343] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 15:29:37 (1787599777) [ 717.960349] Key type .llcrypt unregistered [ 717.964851] Key type ._llcrypt unregistered [ 724.499582] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 724.629963] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 724.635167] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 724.644064] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 726.471887] Key type ._llcrypt registered [ 726.478218] Key type .llcrypt registered [ 726.816583] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 726.834644] alg: No test for adler32 (adler32-zlib) [ 727.880250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 727.968380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 728.016732] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 729.032583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 729.061705] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 732.244640] LNet: 38100:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 732.251874] LNetError: 38100:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 732.260612] LNet: Removed LNI 10.1.2.3@tcp [ 732.269984] LNet: Removed LNI 192.168.203.42@tcp [ 733.068597] Key type .llcrypt unregistered [ 733.078292] Key type ._llcrypt unregistered [ 745.834624] Key type ._llcrypt registered [ 745.836798] Key type .llcrypt registered [ 747.107726] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 15:30:07 (1787599807) [ 747.659230] Key type .llcrypt unregistered [ 747.662828] Key type ._llcrypt unregistered [ 755.198272] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 755.332733] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 755.342949] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 755.352954] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 758.144698] Key type ._llcrypt registered [ 758.146809] Key type .llcrypt registered [ 758.299382] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 758.324367] alg: No test for adler32 (adler32-zlib) [ 759.284821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 759.361946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 759.395873] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 761.793547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 761.837716] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 764.359323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 764.394701] LNet: Removed LNI 10.1.2.3@tcp [ 764.428686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 764.479083] LNet: Removed LNI 10.1.2.31@tcp [ 764.498884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 764.541600] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 765.174686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 765.205928] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 766.458928] LNet: 40027:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 766.465776] LNetError: 40027:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 766.470792] LNet: Removed LNI 10.1.2.31@tcp [ 766.483618] LNet: Removed LNI 10.1.2.3@tcp [ 767.089549] Key type .llcrypt unregistered [ 767.095727] Key type ._llcrypt unregistered [ 779.168800] Key type ._llcrypt registered [ 779.177508] Key type .llcrypt registered [ 780.340299] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 15:30:41 (1787599841) [ 780.780493] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 780.791853] alg: No test for adler32 (adler32-zlib) [ 781.701517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 781.755449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 781.772521] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 782.420732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 782.443265] LNet: 40985:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 782.452280] LNetError: 40985:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 782.473313] LNet: Removed LNI 192.168.203.42@tcp [ 782.488930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 782.532423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.203.42@tcp [ 782.564855] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 783.280366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.203.42@tcp [ 783.322660] LNet: Removed LNI 192.168.203.42@tcp [ 783.457577] LNet: 41121:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 784.105664] Key type .llcrypt unregistered [ 784.109586] Key type ._llcrypt unregistered [ 795.689289] Key type ._llcrypt registered [ 795.693374] Key type .llcrypt registered [ 797.194808] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 15:30:57 (1787599857) [ 798.999583] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg342-server [ 800.918291] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 15:31:01 (1787599861) [ 801.486166] Key type .llcrypt unregistered [ 801.492624] Key type ._llcrypt unregistered [ 807.187391] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 807.368734] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 807.383491] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 807.393927] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 810.066927] Key type ._llcrypt registered [ 810.069538] Key type .llcrypt registered [ 810.297359] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 810.319917] alg: No test for adler32 (adler32-zlib) [ 811.257584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 811.342447] LNet: Added LNI 192.168.203.42@tcp1 [8/256/0/180] [ 811.966762] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 813.882332] LNet: 42965:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 813.901942] LNetError: 42965:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 813.914980] LNet: Removed LNI 192.168.203.42@tcp1 [ 813.935274] LNet: Removed LNI 10.1.2.3@tcp2 [ 814.725357] Key type .llcrypt unregistered [ 814.733285] Key type ._llcrypt unregistered [ 826.431247] Key type ._llcrypt registered [ 826.433255] Key type .llcrypt registered [ 827.932604] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 15:31:28 (1787599888) [ 828.839618] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 828.855231] alg: No test for adler32 (adler32-zlib) [ 829.777960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 829.835244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 829.857988] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 830.514565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 830.530799] LNet: 43942:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 830.535135] LNetError: 43942:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 830.553226] LNet: Removed LNI 192.168.203.42@tcp [ 830.566885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 830.612528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 830.631042] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 837.898173] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 15:31:38 (1787599898) [ 838.068202] LNet: 44525:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 838.079374] LNetError: 44525:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 838.114163] LNet: Removed LNI 192.168.203.42@tcp [ 839.121520] Key type .llcrypt unregistered [ 839.129820] Key type ._llcrypt unregistered [ 847.148941] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 847.281244] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 847.291322] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 847.301770] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 849.912503] Key type ._llcrypt registered [ 849.914807] Key type .llcrypt registered [ 850.127060] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 850.163302] alg: No test for adler32 (adler32-zlib) [ 851.162106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 851.278142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 851.336236] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 852.525369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 852.544879] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 856.010306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 856.071972] LNet: 45275:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 856.086253] LNetError: 45275:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 856.100351] LNet: Removed LNI 10.1.2.3@tcp [ 856.121831] LNet: Removed LNI 192.168.203.42@tcp [ 856.167678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 856.235811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 856.298455] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 856.316128] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 856.813687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 856.831596] LNet: 45302:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 856.839738] LNetError: 45302:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 856.845486] LNetError: 45302:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 856.852610] LNet: Removed LNI 10.1.2.3@tcp [ 856.863656] LNet: Removed LNI 192.168.203.42@tcp [ 856.884761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 856.941912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 856.976519] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 856.984235] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 857.545913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 857.563742] LNet: Added LNI 192.168.203.42@tcp2 [8/256/0/180] [ 858.455481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 858.481872] LNet: 45464:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 858.489872] LNetError: 45464:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 858.496537] LNetError: 45464:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 858.505134] LNet: Removed LNI 10.1.2.3@tcp [ 858.508157] LNet: Removed LNI 192.168.203.42@tcp [ 858.521993] LNet: Removed LNI 192.168.203.42@tcp2 [ 858.553652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 858.659967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 858.698962] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 858.708801] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 858.728514] LNet: Added LNI 192.168.203.42@tcp2 [8/256/0/180] [ 866.892204] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 15:32:07 (1787599927) [ 867.368270] LNet: 46062:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 867.385120] LNetError: 46062:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 867.394593] LNetError: 46062:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 867.413197] LNet: Removed LNI 10.1.2.3@tcp [ 867.417135] LNet: Removed LNI 192.168.203.42@tcp [ 867.438207] LNet: Removed LNI 192.168.203.42@tcp2 [ 868.201943] Key type .llcrypt unregistered [ 868.205666] Key type ._llcrypt unregistered [ 875.406506] Key type ._llcrypt registered [ 875.412513] Key type .llcrypt registered [ 875.656404] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 875.668595] alg: No test for adler32 (adler32-zlib) [ 876.597521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 876.643803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 876.664928] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 877.507496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 877.524593] LNet: 46461:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 877.532330] LNetError: 46461:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 877.548089] LNet: Removed LNI 192.168.203.42@tcp [ 877.591505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 877.706670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 877.742438] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 878.239453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 878.271781] LNet: 46491:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 878.285715] LNetError: 46491:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 878.310466] LNet: Removed LNI 192.168.203.42@tcp [ 878.338874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 878.408597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 878.447346] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 878.977379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 879.021870] LNet: Removed LNI 192.168.203.42@tcp [ 879.068071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 879.153458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 879.193772] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 879.726187] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 879.922491] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 879.937243] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 879.944462] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 882.452473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 882.482221] LNet: 46602:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 882.489468] LNet: 46602:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 882.495848] LNetError: 46602:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 882.502921] LNetError: 46602:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 882.530030] LNet: Removed LNI 192.168.203.42@tcp [ 882.549139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 882.608720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 882.645493] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 889.457295] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 15:32:30 (1787599950) [ 889.650771] LNet: 47194:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 889.667164] LNetError: 47194:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 889.705897] LNet: Removed LNI 192.168.203.42@tcp [ 890.355893] Key type .llcrypt unregistered [ 890.361363] Key type ._llcrypt unregistered [ 895.787875] Key type ._llcrypt registered [ 895.791531] Key type .llcrypt registered [ 895.964765] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 895.977956] alg: No test for adler32 (adler32-zlib) [ 896.897932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 896.965937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 897.008838] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 897.965657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 897.991633] LNet: 47601:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 898.001293] LNetError: 47601:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 898.018721] LNet: Removed LNI 192.168.203.42@tcp [ 898.039564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 898.083411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 898.136468] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 905.302987] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 15:32:46 (1787599966) [ 905.362120] LNet: 48186:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 905.371789] LNetError: 48186:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 905.405277] LNet: Removed LNI 192.168.203.42@tcp [ 906.028062] Key type .llcrypt unregistered [ 906.032270] Key type ._llcrypt unregistered [ 912.657660] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 912.927058] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 912.929863] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 912.947617] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 915.678083] Key type ._llcrypt registered [ 915.681631] Key type .llcrypt registered [ 915.965633] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 916.001860] alg: No test for adler32 (adler32-zlib) [ 916.982421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 917.068447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 917.117158] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 918.160934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 918.198780] LNet: 48648:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 918.208324] LNetError: 48648:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 918.233576] LNet: Removed LNI 192.168.203.42@tcp [ 918.247924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 918.343905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 918.378203] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 925.095220] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 15:33:05 (1787599985) [ 925.197403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 925.235249] LNet: 49238:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 925.245793] LNetError: 49238:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 925.266777] LNet: Removed LNI 192.168.203.42@tcp [ 925.292470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 925.467437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 931.285498] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 15:33:12 (1787599992) [ 931.358705] LNet: 49822:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 931.877127] Key type .llcrypt unregistered [ 931.880484] Key type ._llcrypt unregistered [ 937.477663] Key type ._llcrypt registered [ 937.479942] Key type .llcrypt registered [ 937.668347] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 937.689442] alg: No test for adler32 (adler32-zlib) [ 938.620195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 938.670114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 938.687621] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 939.468074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 939.490917] LNet: 50230:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 939.495411] LNetError: 50230:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 939.513301] LNet: Removed LNI 192.168.203.42@tcp [ 939.527351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 939.585287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 939.624243] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 945.933440] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 15:33:26 (1787600006) [ 946.144574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 946.170773] LNet: 50823:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 946.176333] LNetError: 50823:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 946.193776] LNet: Removed LNI 192.168.203.42@tcp [ 946.230516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 946.283179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 952.406804] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 15:33:33 (1787600013) [ 954.040121] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 956.053409] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 15:33:36 (1787600016) [ 957.961712] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 959.656340] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 15:33:40 (1787600020) [ 960.943130] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 962.916470] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 15:33:43 (1787600023) [ 964.483259] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 966.252124] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 15:33:46 (1787600026) [ 967.683511] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 969.310372] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 15:33:50 (1787600030) [ 969.519313] LNet: 52876:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 970.087670] Key type .llcrypt unregistered [ 970.090880] Key type ._llcrypt unregistered [ 976.223430] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 976.369680] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 976.381943] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 976.394891] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 978.966206] Key type ._llcrypt registered [ 978.968784] Key type .llcrypt registered [ 979.182815] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 979.201206] alg: No test for adler32 (adler32-zlib) [ 980.184272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 980.299029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 980.327651] LNet: Added LNI 192.168.203.42@tcp [8/512/0/180] [ 980.333950] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 980.346536] LNet: 53200:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 980.551332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 980.606450] LNet: 53219:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 980.613336] LNetError: 53219:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 980.623139] LNet: Removed LNI 10.1.2.3@tcp [ 980.627342] LNetError: 53219:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 980.657761] LNet: Removed LNI 192.168.203.42@tcp [ 980.677359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 980.767567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 980.801626] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 980.899949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 980.926418] LNet: 53243:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 980.937533] LNetError: 53243:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 980.971025] LNet: Removed LNI 192.168.203.42@tcp [ 981.011428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 981.081575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 981.118904] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 981.125200] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 981.130621] LNet: 53251:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 981.240644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 981.260483] LNet: 53269:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 981.267613] LNetError: 53269:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 981.275948] LNet: Removed LNI 10.1.2.3@tcp [ 981.277438] LNetError: 53269:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 981.301310] LNet: Removed LNI 192.168.203.42@tcp [ 981.332854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 981.402404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 981.423888] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 981.428307] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 981.431783] LNet: 53277:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 981.624639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 981.642285] LNet: 53303:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 981.649978] LNetError: 53303:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 981.658398] LNet: Removed LNI 10.1.2.3@tcp [ 981.661690] LNetError: 53303:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 981.674729] LNet: Removed LNI 192.168.203.42@tcp [ 981.707309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 981.784331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 981.815655] LNet: Added LNI 192.168.203.42@tcp [8/256/0/182] [ 981.825671] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 981.947494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 981.964648] LNet: 53331:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 981.971688] LNetError: 53331:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 981.979818] LNet: Removed LNI 10.1.2.3@tcp [ 981.982575] LNetError: 53331:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 981.994760] LNet: Removed LNI 192.168.203.42@tcp [ 982.009807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 982.100352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 982.119826] LNet: Added LNI 192.168.203.42@tcp [8/256/0/182] [ 982.126323] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 982.130278] LNet: 53340:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 982.218753] LNet: Removed LNI 10.1.2.3@tcp [ 982.240123] LNet: Removed LNI 192.168.203.42@tcp [ 982.932826] Key type .llcrypt unregistered [ 982.938515] Key type ._llcrypt unregistered [ 994.797714] Key type ._llcrypt registered [ 994.800312] Key type .llcrypt registered [ 996.090147] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 15:34:16 (1787600056) [ 997.734382] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 999.476626] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 15:34:20 (1787600060) [ 1000.072830] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1000.084208] alg: No test for adler32 (adler32-zlib) [ 1001.102622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1001.147673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1001.175703] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 1001.980749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.203.42 [ 1002.018123] LNet: 54614:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1002.026889] LNet: Removed LNI 192.168.203.42@tcp [ 1007.838619] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 15:34:28 (1787600068) [ 1008.322804] LNet: 55185:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1008.339577] LNetError: 55185:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1008.353189] LNet: Removed LNI 192.168.203.42@tcp [ 1008.926656] Key type .llcrypt unregistered [ 1008.928423] Key type ._llcrypt unregistered [ 1015.065540] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1015.262989] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1017.909739] Key type ._llcrypt registered [ 1017.912712] Key type .llcrypt registered [ 1018.128591] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1018.152912] alg: No test for adler32 (adler32-zlib) [ 1019.387232] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1024.516878] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 15:34:45 (1787600085) [ 1024.947827] LNet: 56104:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1024.957699] LNetError: 56104:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1024.977584] LNet: Removed LNI 10.1.2.3@tcp [ 1025.662565] Key type .llcrypt unregistered [ 1025.665167] Key type ._llcrypt unregistered [ 1030.813582] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1031.079960] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1031.085685] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1031.092597] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1033.891947] Key type ._llcrypt registered [ 1033.894439] Key type .llcrypt registered [ 1034.073920] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1034.110816] alg: No test for adler32 (adler32-zlib) [ 1035.197537] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1041.532211] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 15:35:02 (1787600102) [ 1042.899862] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1044.671759] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 15:35:05 (1787600105) [ 1044.877976] LNet: 57292:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1044.887441] LNetError: 57292:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1044.897962] LNet: Removed LNI 10.1.2.3@tcp [ 1045.704546] Key type .llcrypt unregistered [ 1045.711295] Key type ._llcrypt unregistered [ 1051.732964] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1051.924793] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1051.928647] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1051.932367] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1054.653274] Key type ._llcrypt registered [ 1054.660672] Key type .llcrypt registered [ 1054.851778] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1054.868109] alg: No test for adler32 (adler32-zlib) [ 1055.750979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1055.799401] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1063.251919] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 15:35:24 (1787600124) [ 1063.754749] LNet: 58524:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1063.761584] LNetError: 58524:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1063.784433] LNet: Removed LNI 10.1.2.3@tcp [ 1064.420188] Key type .llcrypt unregistered [ 1064.424171] Key type ._llcrypt unregistered [ 1069.676500] Key type ._llcrypt registered [ 1069.677935] Key type .llcrypt registered [ 1069.886170] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1069.909133] alg: No test for adler32 (adler32-zlib) [ 1070.807816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1070.836815] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 1076.989848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.142@tcp [ 1077.445293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.42@tcp -d 192.168.203.142@tcp -m GET -e local_interrupt -r 1 [ 1077.483214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.42@tcp -d 192.168.203.42@tcp -m GET -e local_interrupt -r 1 [ 1077.521579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.142@tcp [ 1077.542980] LNet: There was an unexpected network error while writing to 192.168.203.142: rc = -22 [ 1077.726302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1077.760243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1077.784748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1077.822438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1077.847714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1077.875666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1077.901865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1077.930470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1077.960715] LNet: 58966:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1077.983485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1078.027980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1078.047700] LNet: 1 local NIs in recovery (showing 1): 192.168.203.42@tcp [ 1078.054859] LNet: There was an unexpected network error while writing to 192.168.203.42: rc = -22 [ 1078.066562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1078.091528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1078.137193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1078.170717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1078.208321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1078.693055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.42@tcp -d 192.168.203.142@tcp -m GET -e local_dropped -r 1 [ 1078.731047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.42@tcp -d 192.168.203.42@tcp -m GET -e local_dropped -r 1 [ 1078.769083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.142@tcp [ 1078.936961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1078.966886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1078.992895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1079.018853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1079.042781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1079.075260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1079.112467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1079.138751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1079.166310] LNet: 59075:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1079.195799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1079.237437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1079.263954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1079.291668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1079.323519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1079.354464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1079.388095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1079.794271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.42@tcp -d 192.168.203.142@tcp -m GET -e local_aborted -r 1 [ 1079.827803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.42@tcp -d 192.168.203.42@tcp -m GET -e local_aborted -r 1 [ 1079.854585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.142@tcp [ 1079.867381] LNet: There was an unexpected network error while writing to 192.168.203.142: rc = -22 [ 1079.871363] LNet: Skipped 1 previous similar message [ 1080.073927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1080.116554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1080.147290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1080.178705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1080.201676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1080.228513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1080.256562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1080.288215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1080.308314] LNet: 59184:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1080.333161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1080.358789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1080.385269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1080.411396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1080.436209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1080.466317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1080.494576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1080.904222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.42@tcp -d 192.168.203.142@tcp -m GET -e local_no_route -r 1 [ 1080.938977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.42@tcp -d 192.168.203.42@tcp -m GET -e local_no_route -r 1 [ 1080.963614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.142@tcp [ 1081.140788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1081.166630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1081.197701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1081.220849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1081.247390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1081.272144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1081.297407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1081.332272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1081.360852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1081.391842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1081.432266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1081.460539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1081.491680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1081.520225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1081.547427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1081.898396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.42@tcp -d 192.168.203.142@tcp -m GET -e local_timeout -r 1 [ 1081.935218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.42@tcp -d 192.168.203.42@tcp -m GET -e local_timeout -r 1 [ 1081.984575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.142@tcp [ 1082.013635] LNet: There was an unexpected network error while writing to 192.168.203.142: rc = -22 [ 1082.017464] LNet: Skipped 2 previous similar messages [ 1082.204952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1082.240675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1082.265800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1082.291568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1082.313702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1082.339278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1082.365941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1082.402794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1082.418307] LNet: 59402:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1082.426327] LNet: 59402:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1082.439710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1082.462816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1082.488542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1082.516214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1082.558340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1082.597309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1082.622824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1083.011898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.42@tcp -d 192.168.203.142@tcp -m GET -e local_error -r 1 [ 1083.053616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.42@tcp -d 192.168.203.42@tcp -m GET -e local_error -r 1 [ 1083.124712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.142@tcp [ 1083.350332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1083.383323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1083.405307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1083.452502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1083.501914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1083.548531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1083.600594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1083.632517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1083.668362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1083.702491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1083.731855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1083.763412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1083.809568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1083.838099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1083.861872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1084.141113] LNet: 59557:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1084.150681] LNetError: 59557:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1084.167196] LNet: Removed LNI 192.168.203.42@tcp [ 1084.548533] Key type .llcrypt unregistered [ 1084.551495] Key type ._llcrypt unregistered [ 1099.157794] Key type ._llcrypt registered [ 1099.160696] Key type .llcrypt registered [ 1100.284375] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 15:36:01 (1787600161) [ 1101.068379] Key type .llcrypt unregistered [ 1101.070774] Key type ._llcrypt unregistered [ 1107.144709] Key type ._llcrypt registered [ 1107.152239] Key type .llcrypt registered [ 1107.445868] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1107.463782] alg: No test for adler32 (adler32-zlib) [ 1108.433385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1108.477763] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 1115.537757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.142@tcp [ 1127.117399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1127.176630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1127.214692] LNet: Added LNI 192.168.203.42@tcp1 [8/256/0/180] [ 1129.016902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.42@tcp -d 192.168.203.142@tcp -m GET -e local_interrupt -r 1 [ 1129.051504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.42@tcp -d 192.168.203.142@tcp1 -m GET -e local_interrupt -r 1 [ 1129.087485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.42@tcp -d 192.168.203.42@tcp -m GET -e local_interrupt -r 1 [ 1129.128511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.42@tcp -d 192.168.203.42@tcp1 -m GET -e local_interrupt -r 1 [ 1129.163882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.42@tcp1 -d 192.168.203.142@tcp -m GET -e local_interrupt -r 1 [ 1129.198380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.42@tcp1 -d 192.168.203.142@tcp1 -m GET -e local_interrupt -r 1 [ 1129.231649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.42@tcp1 -d 192.168.203.42@tcp -m GET -e local_interrupt -r 1 [ 1129.279696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.42@tcp1 -d 192.168.203.42@tcp1 -m GET -e local_interrupt -r 1 [ 1129.313571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.142@tcp [ 1129.332685] LNet: There was an unexpected network error while writing to 192.168.203.142: rc = -22 [ 1129.339498] LNet: 1 local NIs in recovery (showing 1): 192.168.203.42@tcp [ 1129.637248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1129.696352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1129.748874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1129.807416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1129.871224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1129.915459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1129.966436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1130.022350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1130.047738] LNet: 61028:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1130.077336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1130.112306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1130.155866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1130.195267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1130.245678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1130.320704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1130.380133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1130.807976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.42@tcp -d 192.168.203.142@tcp -m GET -e local_dropped -r 1 [ 1130.835203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.42@tcp -d 192.168.203.142@tcp1 -m GET -e local_dropped -r 1 [ 1130.862805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.42@tcp -d 192.168.203.42@tcp -m GET -e local_dropped -r 1 [ 1130.894860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.42@tcp -d 192.168.203.42@tcp1 -m GET -e local_dropped -r 1 [ 1130.921161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.42@tcp1 -d 192.168.203.142@tcp -m GET -e local_dropped -r 1 [ 1130.953748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.42@tcp1 -d 192.168.203.142@tcp1 -m GET -e local_dropped -r 1 [ 1130.985506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.42@tcp1 -d 192.168.203.42@tcp -m GET -e local_dropped -r 1 [ 1131.020674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.42@tcp1 -d 192.168.203.42@tcp1 -m GET -e local_dropped -r 1 [ 1131.056190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.142@tcp [ 1131.072696] LNet: There was an unexpected network error while writing to 192.168.203.142: rc = -22 [ 1131.078244] LNet: Skipped 2 previous similar messages [ 1131.290894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1131.321538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1131.350106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1131.390697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1131.436326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1131.506851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1131.543466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1131.564779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1131.579624] LNet: 61149:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1131.598259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1131.643430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1131.708367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1131.755332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1131.809802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1131.855760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1131.896469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1132.323553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.42@tcp -d 192.168.203.142@tcp -m GET -e local_aborted -r 1 [ 1132.385990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.42@tcp -d 192.168.203.142@tcp1 -m GET -e local_aborted -r 1 [ 1132.424945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.42@tcp -d 192.168.203.42@tcp -m GET -e local_aborted -r 1 [ 1132.480646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.42@tcp -d 192.168.203.42@tcp1 -m GET -e local_aborted -r 1 [ 1132.548627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.42@tcp1 -d 192.168.203.142@tcp -m GET -e local_aborted -r 1 [ 1132.618746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.42@tcp1 -d 192.168.203.142@tcp1 -m GET -e local_aborted -r 1 [ 1132.677610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.42@tcp1 -d 192.168.203.42@tcp -m GET -e local_aborted -r 1 [ 1132.724965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.42@tcp1 -d 192.168.203.42@tcp1 -m GET -e local_aborted -r 1 [ 1132.770236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.142@tcp [ 1132.790852] LNet: There was an unexpected network error while writing to 192.168.203.142: rc = -22 [ 1132.804885] LNet: Skipped 2 previous similar messages [ 1133.105537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1133.124377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1133.142877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1133.168758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1133.190029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1133.215679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1133.245615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1133.275345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1133.294228] LNet: 61271:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1133.316908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1133.339919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1133.389158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1133.434053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1133.458405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1133.511934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1133.538757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1134.179768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.42@tcp -d 192.168.203.142@tcp -m GET -e local_no_route -r 1 [ 1134.214248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.42@tcp -d 192.168.203.142@tcp1 -m GET -e local_no_route -r 1 [ 1134.249207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.42@tcp -d 192.168.203.42@tcp -m GET -e local_no_route -r 1 [ 1134.284396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.42@tcp -d 192.168.203.42@tcp1 -m GET -e local_no_route -r 1 [ 1134.319844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.42@tcp1 -d 192.168.203.142@tcp -m GET -e local_no_route -r 1 [ 1134.353190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.42@tcp1 -d 192.168.203.142@tcp1 -m GET -e local_no_route -r 1 [ 1134.383317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.42@tcp1 -d 192.168.203.42@tcp -m GET -e local_no_route -r 1 [ 1134.424334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.42@tcp1 -d 192.168.203.42@tcp1 -m GET -e local_no_route -r 1 [ 1134.465334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.142@tcp [ 1134.764963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1134.816405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1134.884797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1134.947116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1134.989984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1135.033326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1135.069248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1135.124805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1135.209552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1135.297917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1135.370071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1135.421321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1135.484158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1135.524637] LNet: There was an unexpected network error while writing to 192.168.203.42: rc = -22 [ 1135.531566] LNet: Skipped 6 previous similar messages [ 1135.532303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1135.608361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1136.232576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.42@tcp -d 192.168.203.142@tcp -m GET -e local_timeout -r 1 [ 1136.292704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.42@tcp -d 192.168.203.142@tcp1 -m GET -e local_timeout -r 1 [ 1136.340941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.42@tcp -d 192.168.203.42@tcp -m GET -e local_timeout -r 1 [ 1136.385234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.42@tcp -d 192.168.203.42@tcp1 -m GET -e local_timeout -r 1 [ 1136.427856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.42@tcp1 -d 192.168.203.142@tcp -m GET -e local_timeout -r 1 [ 1136.458569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.42@tcp1 -d 192.168.203.142@tcp1 -m GET -e local_timeout -r 1 [ 1136.516769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.42@tcp1 -d 192.168.203.42@tcp -m GET -e local_timeout -r 1 [ 1136.587773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.42@tcp1 -d 192.168.203.42@tcp1 -m GET -e local_timeout -r 1 [ 1136.643284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.142@tcp [ 1136.970108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1137.057717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1137.104097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1137.166186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1137.210958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1137.243173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1137.276737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1137.312412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1137.340231] LNet: 61513:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1137.345252] LNet: 61513:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1137.358966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1137.392710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1137.426655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1137.468560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1137.496609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1137.530168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1137.587986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1138.266431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.42@tcp -d 192.168.203.142@tcp -m GET -e local_error -r 1 [ 1138.294958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.42@tcp -d 192.168.203.142@tcp1 -m GET -e local_error -r 1 [ 1138.322763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.42@tcp -d 192.168.203.42@tcp -m GET -e local_error -r 1 [ 1138.351653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.42@tcp -d 192.168.203.42@tcp1 -m GET -e local_error -r 1 [ 1138.419638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.42@tcp1 -d 192.168.203.142@tcp -m GET -e local_error -r 1 [ 1138.499537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.42@tcp1 -d 192.168.203.142@tcp1 -m GET -e local_error -r 1 [ 1138.557144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.42@tcp1 -d 192.168.203.42@tcp -m GET -e local_error -r 1 [ 1138.601885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.42@tcp1 -d 192.168.203.42@tcp1 -m GET -e local_error -r 1 [ 1138.631842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.142@tcp [ 1138.983152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1139.069877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1139.104993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1139.181100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1139.243787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1139.278184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1139.343323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1139.401789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1139.430816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1139.549652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1139.587660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1139.679620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1139.778898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1139.830684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1139.891518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1145.333067] LNet: 61714:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1145.340752] LNetError: 61714:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1145.348227] LNet: Removed LNI 192.168.203.42@tcp [ 1146.422423] LNet: Removed LNI 192.168.203.42@tcp1 [ 1146.956300] Key type .llcrypt unregistered [ 1146.959389] Key type ._llcrypt unregistered [ 1163.791951] Key type ._llcrypt registered [ 1163.800669] Key type .llcrypt registered [ 1164.903272] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 15:37:05 (1787600225) [ 1165.916369] Key type .llcrypt unregistered [ 1165.918258] Key type ._llcrypt unregistered [ 1172.480866] Key type ._llcrypt registered [ 1172.484939] Key type .llcrypt registered [ 1172.698196] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1172.730291] alg: No test for adler32 (adler32-zlib) [ 1173.719782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1173.752545] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 1180.804876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.142@tcp [ 1181.692689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.42@tcp -d 192.168.203.142@tcp -m GET -e remote_dropped -r 1 [ 1181.758220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.42@tcp -d 192.168.203.42@tcp -m GET -e remote_dropped -r 1 [ 1181.843095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.142@tcp [ 1181.858850] LNet: There was an unexpected network error while writing to 192.168.203.142: rc = -22 [ 1182.203300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1182.241673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1182.268491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1182.297825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1182.364896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1182.420866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1182.462815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1182.509230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1182.540089] LNet: 62967:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1182.572689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1182.619437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1182.690956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1182.740043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1182.804902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1182.856258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1182.909917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1183.631540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.42@tcp -d 192.168.203.142@tcp -m GET -e remote_error -r 1 [ 1183.663318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.42@tcp -d 192.168.203.42@tcp -m GET -e remote_error -r 1 [ 1183.719083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.142@tcp [ 1183.745369] LNet: There was an unexpected network error while writing to 192.168.203.142: rc = -22 [ 1184.024343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1184.079315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1184.153140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1184.220569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1184.266758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1184.292610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1184.323125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1184.373896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1184.411253] LNet: 63076:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1184.455559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1184.509064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1184.570143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1184.611242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1184.649305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1184.686892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1184.733552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1185.257351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.42@tcp -d 192.168.203.142@tcp -m GET -e remote_timeout -r 1 [ 1185.281670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.42@tcp -d 192.168.203.42@tcp -m GET -e remote_timeout -r 1 [ 1185.306917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.142@tcp [ 1185.328475] LNet: There was an unexpected network error while writing to 192.168.203.142: rc = -22 [ 1185.530294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1185.560450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1185.584925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1185.611799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1185.633650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1185.662297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1185.711892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1185.738972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1185.754179] LNet: 63185:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1185.767478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1185.800311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1185.839633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1185.888867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1185.938568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1185.990965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1186.062856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1186.323561] LNet: 63231:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1186.328116] LNetError: 63231:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1186.340062] LNet: Removed LNI 192.168.203.42@tcp [ 1186.680648] Key type .llcrypt unregistered [ 1186.684291] Key type ._llcrypt unregistered [ 1200.105441] Key type ._llcrypt registered [ 1200.107752] Key type .llcrypt registered [ 1201.332464] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 15:37:42 (1787600262) [ 1202.058332] Key type .llcrypt unregistered [ 1202.062549] Key type ._llcrypt unregistered [ 1207.577645] Key type ._llcrypt registered [ 1207.579967] Key type .llcrypt registered [ 1207.862042] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1207.885538] alg: No test for adler32 (adler32-zlib) [ 1208.858361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1208.887098] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 1215.718481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.142@tcp [ 1224.194779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1224.264130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1224.285380] LNet: Added LNI 192.168.203.42@tcp1 [8/256/0/180] [ 1225.860455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.42@tcp -d 192.168.203.142@tcp -m GET -e remote_dropped -r 1 [ 1225.952211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.42@tcp -d 192.168.203.142@tcp1 -m GET -e remote_dropped -r 1 [ 1226.016843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.42@tcp -d 192.168.203.42@tcp -m GET -e remote_dropped -r 1 [ 1226.063566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.42@tcp -d 192.168.203.42@tcp1 -m GET -e remote_dropped -r 1 [ 1226.143874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.42@tcp1 -d 192.168.203.142@tcp -m GET -e remote_dropped -r 1 [ 1226.200246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.42@tcp1 -d 192.168.203.142@tcp1 -m GET -e remote_dropped -r 1 [ 1226.260554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.42@tcp1 -d 192.168.203.42@tcp -m GET -e remote_dropped -r 1 [ 1226.318000] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.42@tcp1 -d 192.168.203.42@tcp1 -m GET -e remote_dropped -r 1 [ 1226.418767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.142@tcp [ 1226.463275] LNet: There was an unexpected network error while writing to 192.168.203.142: rc = -22 [ 1226.471199] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.142@tcp [ 1226.805742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1226.833697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1226.886655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1226.921665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1226.977302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1227.028374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1227.070881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1227.099502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1227.135090] LNet: 64700:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1227.158441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1227.210678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1227.260663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1227.308834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1227.349549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1227.398598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1227.452570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1227.516742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1228.236867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.42@tcp -d 192.168.203.142@tcp -m GET -e remote_error -r 1 [ 1228.302960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.42@tcp -d 192.168.203.142@tcp1 -m GET -e remote_error -r 1 [ 1228.361449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.42@tcp -d 192.168.203.42@tcp -m GET -e remote_error -r 1 [ 1228.425739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.42@tcp -d 192.168.203.42@tcp1 -m GET -e remote_error -r 1 [ 1228.476147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.42@tcp1 -d 192.168.203.142@tcp -m GET -e remote_error -r 1 [ 1228.517834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.42@tcp1 -d 192.168.203.142@tcp1 -m GET -e remote_error -r 1 [ 1228.558722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.42@tcp1 -d 192.168.203.42@tcp -m GET -e remote_error -r 1 [ 1228.603811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.42@tcp1 -d 192.168.203.42@tcp1 -m GET -e remote_error -r 1 [ 1228.648639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.142@tcp [ 1228.671554] LNet: There was an unexpected network error while writing to 192.168.203.142: rc = -22 [ 1228.676314] LNet: Skipped 2 previous similar messages [ 1228.944793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1228.982829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1229.017791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1229.055629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1229.078137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1229.105518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1229.130832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1229.167373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1229.188564] LNet: 64824:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1229.212179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1229.250344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1229.282298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1229.320518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1229.354767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1229.390649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1229.435138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1229.485820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1230.048621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.42@tcp -d 192.168.203.142@tcp -m GET -e remote_timeout -r 1 [ 1230.099808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.42@tcp -d 192.168.203.142@tcp1 -m GET -e remote_timeout -r 1 [ 1230.150583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.42@tcp -d 192.168.203.42@tcp -m GET -e remote_timeout -r 1 [ 1230.195625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.42@tcp -d 192.168.203.42@tcp1 -m GET -e remote_timeout -r 1 [ 1230.232581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.42@tcp1 -d 192.168.203.142@tcp -m GET -e remote_timeout -r 1 [ 1230.287345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.42@tcp1 -d 192.168.203.142@tcp1 -m GET -e remote_timeout -r 1 [ 1230.322463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.42@tcp1 -d 192.168.203.42@tcp -m GET -e remote_timeout -r 1 [ 1230.355926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.42@tcp1 -d 192.168.203.42@tcp1 -m GET -e remote_timeout -r 1 [ 1230.386882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.142@tcp [ 1230.405763] LNet: There was an unexpected network error while writing to 192.168.203.142: rc = -22 [ 1230.638090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1230.672621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1230.713377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1230.761908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1230.801582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1230.850847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1230.892395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1230.948408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1230.988400] LNet: 64947:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1231.023765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1231.076330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1231.122459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1231.172737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1231.256678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1231.335655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1231.404883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1231.473164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1236.569337] LNet: 65029:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1236.574681] LNetError: 65029:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1236.580491] LNet: Removed LNI 192.168.203.42@tcp [ 1236.591964] LNet: Removed LNI 192.168.203.42@tcp1 [ 1237.033918] Key type .llcrypt unregistered [ 1237.038874] Key type ._llcrypt unregistered [ 1252.032576] Key type ._llcrypt registered [ 1252.035898] Key type .llcrypt registered [ 1253.227207] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 15:38:34 (1787600314) [ 1254.729957] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1256.443430] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 15:38:37 (1787600317) [ 1257.473451] Key type .llcrypt unregistered [ 1257.477587] Key type ._llcrypt unregistered [ 1263.897276] Key type ._llcrypt registered [ 1263.904035] Key type .llcrypt registered [ 1264.166163] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1264.190725] alg: No test for adler32 (adler32-zlib) [ 1265.132146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1265.175912] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 1273.437910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.142@tcp [ 1274.358481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.42@tcp -d 192.168.203.142@tcp -m GET -e network_timeout -r 1 [ 1274.409883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.42@tcp -d 192.168.203.42@tcp -m GET -e network_timeout -r 1 [ 1274.453542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.142@tcp [ 1274.489642] LNet: There was an unexpected network error while writing to 192.168.203.142: rc = -22 [ 1274.776067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1274.826860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1274.886347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1274.967617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1275.005762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1275.041680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1275.086762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1275.129255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1275.165244] LNet: 66588:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1275.186403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1275.240240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1275.281132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1275.345103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1275.389832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1275.424340] LNet: 1 local NIs in recovery (showing 1): 192.168.203.42@tcp [ 1275.428818] LNet: There was an unexpected network error while writing to 192.168.203.42: rc = -22 [ 1275.460818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1275.865870] LNet: 66631:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1275.880845] LNetError: 66631:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1275.903115] LNet: Removed LNI 192.168.203.42@tcp [ 1276.369786] Key type .llcrypt unregistered [ 1276.373712] Key type ._llcrypt unregistered [ 1293.709547] Key type ._llcrypt registered [ 1293.712290] Key type .llcrypt registered [ 1294.050683] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1294.086552] alg: No test for adler32 (adler32-zlib) [ 1295.026538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1295.064807] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 1302.254936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.142@tcp [ 1310.633877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1310.685690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1310.721195] LNet: Added LNI 192.168.203.42@tcp1 [8/256/0/180] [ 1312.029641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.42@tcp -d 192.168.203.142@tcp -m GET -e network_timeout -r 1 [ 1312.058732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.42@tcp -d 192.168.203.142@tcp1 -m GET -e network_timeout -r 1 [ 1312.089243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.42@tcp -d 192.168.203.42@tcp -m GET -e network_timeout -r 1 [ 1312.116243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.42@tcp -d 192.168.203.42@tcp1 -m GET -e network_timeout -r 1 [ 1312.142948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.42@tcp1 -d 192.168.203.142@tcp -m GET -e network_timeout -r 1 [ 1312.170806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.42@tcp1 -d 192.168.203.142@tcp1 -m GET -e network_timeout -r 1 [ 1312.210974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.42@tcp1 -d 192.168.203.42@tcp -m GET -e network_timeout -r 1 [ 1312.244466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.42@tcp1 -d 192.168.203.42@tcp1 -m GET -e network_timeout -r 1 [ 1312.316619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.142@tcp [ 1312.332315] LNet: There was an unexpected network error while writing to 192.168.203.142: rc = -22 [ 1312.479888] LNet: 1 local NIs in recovery (showing 1): 192.168.203.42@tcp [ 1312.486418] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.142@tcp [ 1312.575232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1312.625955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1312.682764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1312.741626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1312.773550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1312.823978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1312.863413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1312.907453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1312.924562] LNet: 67513:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1312.942578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1312.975918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1313.017834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1313.058974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1313.092291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1313.121039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1318.070598] LNet: There was an unexpected network error while writing to 192.168.203.142: rc = -22 [ 1318.504676] LNet: 67591:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1318.511789] LNetError: 67591:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1319.519216] LNet: Removed LNI 192.168.203.42@tcp [ 1319.526322] LNetError: 67591:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1319.557597] LNet: Removed LNI 192.168.203.42@tcp1 [ 1320.038414] Key type .llcrypt unregistered [ 1320.048778] Key type ._llcrypt unregistered [ 1335.327936] Key type ._llcrypt registered [ 1335.330538] Key type .llcrypt registered [ 1336.568133] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 15:39:57 (1787600397) [ 1336.951438] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1336.964166] alg: No test for adler32 (adler32-zlib) [ 1337.930832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1337.999549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1338.031760] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 1338.626112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1338.665166] LNet: Added LNI 192.168.203.42@tcp1 [8/256/0/180] [ 1339.434674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.42@tcp [ 1339.491401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1339.549720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1339.594610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1339.655277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1339.704724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1339.751591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.42@tcp [ 1339.793223] LNetError: 68690:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.42@tcp to recovery [ 1339.999408] LNet: 1 local NIs in recovery (showing 1): 192.168.203.42@tcp [ 1339.999607] LNet: There was an unexpected network error while writing to 192.168.203.42: rc = -22 [ 1341.024711] LNet: There was an unexpected network error while writing to 192.168.203.42: rc = -22 [ 1342.226196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1345.119800] LNet: There was an unexpected network error while writing to 192.168.203.42: rc = -22 [ 1345.819088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1345.863413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1345.939513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1345.973818] LNet: 68787:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1345.980329] LNetError: 68787:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1345.986463] LNet: Removed LNI 192.168.203.42@tcp [ 1345.998313] LNet: Removed LNI 192.168.203.42@tcp1 [ 1346.040365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1346.100571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1346.124526] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 1346.973826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1347.003337] LNet: Added LNI 192.168.203.42@tcp1 [8/256/0/180] [ 1347.986916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.42@tcp [ 1348.015958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1348.066164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1348.105907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1348.145852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1348.180159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1348.222665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1348.250452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.42@tcp [ 1348.274296] LNetError: 69059:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.42@tcp to recovery [ 1349.153301] LNet: There was an unexpected network error while writing to 192.168.203.42: rc = -22 [ 1351.712580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1355.295629] LNet: There was an unexpected network error while writing to 192.168.203.42: rc = -22 [ 1355.312951] LNet: Skipped 1 previous similar message [ 1359.568883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1359.618465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1359.657317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1365.474750] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 15:40:26 (1787600426) [ 1365.657773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1365.695118] LNet: 69777:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1365.702376] LNetError: 69777:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1365.710669] LNetError: 69777:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1365.722244] LNet: Removed LNI 192.168.203.42@tcp [ 1365.742819] LNet: Removed LNI 192.168.203.42@tcp1 [ 1365.784044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1365.863226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1365.896982] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 1366.574777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1366.599545] LNet: Added LNI 192.168.203.42@tcp1 [8/256/0/180] [ 1367.409354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.42@tcp [ 1367.496877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1367.526978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1367.575833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1367.625716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1367.673243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1367.708682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.42@tcp --health 0 [ 1367.811343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1367.904105] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.42@tcp [ 1368.929125] LNet: There was an unexpected network error while writing to 192.168.203.42: rc = -22 [ 1368.934320] LNet: Skipped 1 previous similar message [ 1383.244827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1383.431901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1383.466695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1383.495576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.42@tcp --health 500 [ 1383.649750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1390.440933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1390.501415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1390.523132] LNet: 70269:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1390.532193] LNetError: 70269:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1390.540336] LNetError: 70269:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1391.583270] LNet: Removed LNI 192.168.203.42@tcp [ 1391.598520] LNet: Removed LNI 192.168.203.42@tcp1 [ 1391.648800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1391.740378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1391.792173] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 1392.901579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1392.941249] LNet: Added LNI 192.168.203.42@tcp1 [8/256/0/180] [ 1393.801525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.42@tcp [ 1393.853243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1393.935601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1393.964706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1394.035633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1394.093924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1394.140455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1394.179288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.42@tcp --health 0 [ 1395.807987] LNet: There was an unexpected network error while writing to 192.168.203.42: rc = -22 [ 1395.821495] LNet: Skipped 3 previous similar messages [ 1396.710331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1406.056380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1406.094495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1406.132485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1412.137800] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 15:41:13 (1787600473) [ 1412.547119] LNet: 71317:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1412.551996] LNetError: 71317:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1412.558502] LNetError: 71317:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1412.567894] LNet: Removed LNI 192.168.203.42@tcp [ 1412.583117] LNet: Removed LNI 192.168.203.42@tcp1 [ 1413.244801] Key type .llcrypt unregistered [ 1413.247774] Key type ._llcrypt unregistered [ 1419.274309] Key type ._llcrypt registered [ 1419.276864] Key type .llcrypt registered [ 1419.517802] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1419.533659] alg: No test for adler32 (adler32-zlib) [ 1420.447940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1420.480157] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 1428.749209] Lustre: DEBUG MARKER: Initial discovery [ 1428.885963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.142@tcp [ 1431.039304] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1431.147284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1431.195452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.142@tcp [ 1431.239223] LNet: There was an unexpected network error while writing to 192.168.203.142: rc = -22 [ 1431.254034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1431.292749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.203.142@tcp -r 1 -m GET -l 3 [ 1431.340531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.203.142@tcp -r 1 -m GET -e local_error [ 1431.379336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.203.142@tcp -r 1 -m PUT -l 6 [ 1431.775359] LNet: 1 local NIs in recovery (showing 1): 192.168.203.42@tcp [ 1432.611332] Lustre: DEBUG MARKER: Force oleg342-client.virtnet to discover 192.168.203.142@tcp (in background) [ 1432.709944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.142@tcp [ 1433.898371] Lustre: DEBUG MARKER: Force oleg342-server to discover 192.168.203.42@tcp [ 1435.743365] LNet: There was an unexpected network error while writing to 192.168.203.142: rc = -22 [ 1436.277871] Lustre: DEBUG MARKER: Wait for 72079 [ 1437.800737] Lustre: DEBUG MARKER: Finished wait on 72079 [ 1437.861671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1437.893883] LNet: There was an unexpected network error while writing to 192.168.203.142: rc = -22 [ 1437.921988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1438.350368] LNet: 72537:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1438.362552] LNetError: 72537:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1438.381769] LNet: Removed LNI 192.168.203.42@tcp [ 1438.780459] Key type .llcrypt unregistered [ 1438.786556] Key type ._llcrypt unregistered [ 1452.054102] Key type ._llcrypt registered [ 1452.056922] Key type .llcrypt registered [ 1453.291258] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 15:41:54 (1787600514) [ 1453.822272] Key type .llcrypt unregistered [ 1453.825045] Key type ._llcrypt unregistered [ 1460.255866] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1460.381184] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1460.391162] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1460.396375] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1462.913951] Key type ._llcrypt registered [ 1462.916434] Key type .llcrypt registered [ 1463.065141] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1463.075971] alg: No test for adler32 (adler32-zlib) [ 1463.948504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1463.993450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1464.013727] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 1464.514218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1464.534585] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1472.048751] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 15:42:12 (1787600532) [ 1472.118438] LNet: 74610:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1472.125497] LNetError: 74610:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1472.133386] LNet: Removed LNI 10.1.2.3@tcp [ 1472.157727] LNet: Removed LNI 192.168.203.42@tcp [ 1472.764500] Key type .llcrypt unregistered [ 1472.771098] Key type ._llcrypt unregistered [ 1478.252554] Key type ._llcrypt registered [ 1478.254178] Key type .llcrypt registered [ 1478.438266] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1478.458144] alg: No test for adler32 (adler32-zlib) [ 1479.408743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1479.479067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1479.528767] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 1480.350728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1480.368403] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1481.193659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.203.42@tcp [ 1482.272241] LNetError: Unexpected error -22 connecting to 192.168.203.42@tcp at host 192.168.203.42:988 [ 1486.946648] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 15:42:28 (1787600548) [ 1487.219709] LNet: 75755:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1487.230618] LNetError: 75755:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1487.238326] LNet: Removed LNI 10.1.2.3@tcp [ 1487.246860] LNet: Removed LNI 192.168.203.42@tcp [ 1487.854846] Key type .llcrypt unregistered [ 1487.858257] Key type ._llcrypt unregistered [ 1493.693565] Key type ._llcrypt registered [ 1493.695854] Key type .llcrypt registered [ 1493.871577] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1493.891392] alg: No test for adler32 (adler32-zlib) [ 1494.810342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1494.871766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1494.901965] LNet: Added LNI 192.168.203.42@tcp1 [8/256/0/180] [ 1495.480557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1495.497817] LNet: Added LNI 192.168.203.42@tcp2 [8/256/0/180] [ 1496.228266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.203.42@tcp1 --nid 192.168.203.42@tcp2 [ 1499.770370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.42@tcp1 192.168.203.42@tcp2 [ 1499.797860] LNetError: 76595:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.203.42@tcp1 to 192.168.203.42@tcp2 - Route table may be misconfigured [ 1499.834358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.42@tcp2 192.168.203.42@tcp1 [ 1505.433329] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 15:42:46 (1787600566) [ 1505.528243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1505.555830] LNet: 77162:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1505.566788] LNetError: 77162:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1505.572300] LNet: Removed LNI 192.168.203.42@tcp1 [ 1506.612162] LNet: Removed LNI 192.168.203.42@tcp2 [ 1506.641529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1506.698108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1506.725549] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 1507.361518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1507.381876] LNet: Added LNI 192.168.203.42@tcp1 [8/256/0/180] [ 1508.012715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.42@tcp [ 1508.048643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.42@tcp 192.168.203.42@tcp [ 1508.086641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.42@tcp1 192.168.203.42@tcp1 [ 1508.124543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.42@tcp -d 192.168.203.42@tcp -e network_timeout [ 1508.166842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.42@tcp -d 192.168.203.42@tcp1 -e network_timeout [ 1508.206111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.42@tcp1 -d 192.168.203.42@tcp -e network_timeout [ 1508.234879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.42@tcp1 -d 192.168.203.42@tcp1 -e network_timeout [ 1508.282132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.42@tcp [ 1508.300218] LNet: There was an unexpected network error while writing to 192.168.203.42: rc = -22 [ 1508.313755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1508.404481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1508.503252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1508.704098] LNet: 1 local NIs in recovery (showing 1): 192.168.203.42@tcp [ 1513.899078] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 15:42:54 (1787600574) [ 1513.976858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1514.002675] LNet: 78023:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1514.009934] LNetError: 78023:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1514.016269] LNetError: 78023:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1514.021931] LNet: Removed LNI 192.168.203.42@tcp [ 1515.049571] LNet: Removed LNI 192.168.203.42@tcp1 [ 1515.078213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1515.175972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1515.424768] LNet: 78071:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1515.796766] Key type .llcrypt unregistered [ 1515.798651] Key type ._llcrypt unregistered [ 1525.575629] Key type ._llcrypt registered [ 1525.577435] Key type .llcrypt registered [ 1526.723094] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 15:43:07 (1787600587) [ 1527.039861] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1527.060104] alg: No test for adler32 (adler32-zlib) [ 1528.025806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1529.342874] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1531.097099] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 15:43:11 (1787600591) [ 1531.332349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1531.347968] LNet: 79169:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1531.366535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1531.408603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1531.432980] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 1532.013655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1532.030648] LNet: Added LNI 192.168.203.42@tcp1 [8/256/0/180] [ 1532.936720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.42@tcp [ 1532.977387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.42@tcp1 [ 1533.015706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.42@tcp1 [ 1539.246510] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 15:43:19 (1787600599) [ 1541.273205] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg342-server [ 1543.018060] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 15:43:23 (1787600603) [ 1544.724713] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg342-server [ 1546.474064] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 15:43:27 (1787600607) [ 1548.217975] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg342-server [ 1550.047435] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 15:43:30 (1787600610) [ 1552.060878] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg342-server [ 1554.741603] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 15:43:34 (1787600614) [ 1557.879260] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg342-server [ 1560.011401] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 15:43:40 (1787600620) [ 1562.304234] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg342-server [ 1564.737146] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 15:43:44 (1787600624) [ 1568.194780] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg342-server [ 1570.170279] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 15:43:50 (1787600630) [ 1572.453600] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg342-server [ 1574.467149] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 15:43:54 (1787600634) [ 1576.241839] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg342-server [ 1578.246514] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 15:43:58 (1787600638) [ 1578.603855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1578.640731] LNet: 83048:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1578.656312] LNetError: 83048:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1578.671167] LNet: Removed LNI 192.168.203.42@tcp [ 1579.689392] LNet: Removed LNI 192.168.203.42@tcp1 [ 1579.753710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1579.833849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1579.871747] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 1580.950832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1581.087821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.42@tcp [ 1581.289317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1581.306241] LNet: 83196:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1581.312122] LNetError: 83196:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1581.316417] LNetError: 83196:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1581.327551] LNet: Removed LNI 192.168.203.42@tcp [ 1581.352243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1581.403193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1581.433173] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 1582.417101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.42@tcp [ 1582.673731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1582.694668] LNetError: 83341:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1582.708332] LNet: Removed LNI 192.168.203.42@tcp [ 1582.722952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1582.769778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1582.789851] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 1583.641946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1583.714901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.42@tcp [ 1583.899257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1583.922161] LNet: 83488:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1583.928155] LNet: 83488:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1585.002773] LNet: Removed LNI 192.168.203.42@tcp [ 1585.077889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1585.215882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1585.250731] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 1585.999479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.42@tcp [ 1586.219338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1586.242811] LNetError: 83634:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1586.247947] LNetError: 83634:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1586.270971] LNet: Removed LNI 192.168.203.42@tcp [ 1586.301151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1586.376658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1586.401322] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 1587.271499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1587.371288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.42@tcp [ 1587.623779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1588.713144] LNet: Removed LNI 192.168.203.42@tcp [ 1588.757033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1588.830089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1588.878718] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 1589.658851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.42@tcp [ 1589.903782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1589.922399] LNet: 83926:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1589.927719] LNet: 83926:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1590.956378] LNet: Removed LNI 192.168.203.42@tcp [ 1590.998159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1591.075919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1591.096068] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 1591.728013] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1591.792631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.42@tcp [ 1591.996410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1592.028596] LNetError: 84074:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1592.034707] LNetError: 84074:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1592.059193] LNet: Removed LNI 192.168.203.42@tcp [ 1592.085146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1592.172547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1592.201313] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 1592.998665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.42@tcp [ 1593.249371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1594.279671] LNet: Removed LNI 192.168.203.42@tcp [ 1594.306237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1594.372238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1594.397707] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 1595.036949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1595.107641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.42@tcp [ 1595.373192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1596.457965] LNet: Removed LNI 192.168.203.42@tcp [ 1596.495208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1596.554995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1596.585893] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 1597.144575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.42@tcp [ 1598.486265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1598.556676] LNet: 84520:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1598.562243] LNet: 84520:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1599.595542] LNet: Removed LNI 192.168.203.42@tcp [ 1599.640077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1599.713211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1599.742248] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 1600.599856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1600.686469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.42@tcp [ 1600.971638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1600.999898] LNetError: 84668:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1601.005827] LNetError: 84668:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1601.045438] LNet: Removed LNI 192.168.203.42@tcp [ 1601.068987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1601.123658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1601.168331] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 1601.853040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.42@tcp [ 1602.134705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1603.247431] LNet: Removed LNI 192.168.203.42@tcp [ 1603.279495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1603.373302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1603.428364] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 1604.272505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1604.331980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1604.366419] LNet: Removed LNI 192.168.203.42@tcp [ 1604.387935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1604.467503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1604.504183] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 1605.263525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1605.338310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1605.360852] LNet: Removed LNI 192.168.203.42@tcp [ 1605.377670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1605.432590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1605.512680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1605.552255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1605.621913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1605.723741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1605.763412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1605.833629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1605.902948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1605.933857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1605.973143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1606.000559] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 1606.734131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1606.791771] LNet: Removed LNI 192.168.203.42@tcp [ 1606.827747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1606.920917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1613.725995] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 15:44:34 (1787600674) [ 1614.022665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1614.105095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1614.185585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1614.209841] LNet: Added LNI 192.168.203.42@tcp231 [8/256/0/180] [ 1615.678551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1615.709551] LNet: 85972:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1615.724282] LNet: 85972:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 1615.750140] LNet: Removed LNI 192.168.203.42@tcp231 [ 1615.774915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1615.822737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1615.842515] LNet: Added LNI 192.168.203.42@tcp231 [8/256/0/180] [ 1615.846484] LNet: 85980:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1616.343636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1616.378823] LNet: Removed LNI 192.168.203.42@tcp231 [ 1616.411093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1616.441780] LNet: Added LNI 192.168.203.42@tcp231 [8/256/0/180] [ 1617.601820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1617.618517] LNetError: 86124:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1617.623924] LNetError: 86124:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1617.636705] LNet: Removed LNI 192.168.203.42@tcp231 [ 1617.656831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1617.726928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1617.754944] LNet: Added LNI 192.168.203.42@tcp231 [8/256/0/0] [ 1617.758253] LNet: 86132:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1618.222229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1618.266736] LNet: Removed LNI 192.168.203.42@tcp231 [ 1618.291448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1618.324979] LNet: Added LNI 192.168.203.42@tcp231 [8/256/0/0] [ 1619.402568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1619.432281] LNet: Removed LNI 192.168.203.42@tcp231 [ 1619.458711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1619.562429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1619.596137] LNet: Added LNI 192.168.203.42@tcp231 [8/256/0/60] [ 1619.602623] LNet: 86284:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1620.314604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1620.383433] LNet: Removed LNI 192.168.203.42@tcp231 [ 1620.406541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1620.445684] LNet: Added LNI 192.168.203.42@tcp231 [8/256/0/60] [ 1621.697750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1621.738688] LNet: Removed LNI 192.168.203.42@tcp231 [ 1621.777341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1621.861798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1621.897106] LNet: Added LNI 192.168.203.42@tcp231 [8/256/0/180] [ 1621.903427] LNet: 86435:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1622.297949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1622.360574] LNet: Removed LNI 192.168.203.42@tcp231 [ 1622.388766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1622.431846] LNet: Added LNI 192.168.203.42@tcp231 [8/256/0/180] [ 1629.891077] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 15:44:50 (1787600690) [ 1630.068698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1630.109141] LNet: Removed LNI 192.168.203.42@tcp231 [ 1630.126865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1630.233346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1630.269871] LNet: Added LNI 192.168.203.42@tcp232 [8/256/0/180] [ 1631.013122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1631.074295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1631.113867] LNet: Removed LNI 192.168.203.42@tcp232 [ 1631.137617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1631.160454] LNet: Added LNI 192.168.203.42@tcp232 [8/256/0/180] [ 1631.796781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1631.845520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1631.865257] LNet: Removed LNI 192.168.203.42@tcp232 [ 1631.876430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1631.915515] LNet: Added LNI 192.168.203.42@tcp232 [8/256/0/180] [ 1632.534012] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1632.572670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1632.601740] LNet: Removed LNI 192.168.203.42@tcp232 [ 1632.619474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1632.640432] LNet: Added LNI 192.168.203.42@tcp232 [8/256/0/180] [ 1633.184395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1633.211281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1633.245263] LNet: Removed LNI 192.168.203.42@tcp232 [ 1633.262719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1638.255551] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 15:44:59 (1787600699) [ 1639.206508] Key type .llcrypt unregistered [ 1639.211787] Key type ._llcrypt unregistered [ 1644.112251] Key type ._llcrypt registered [ 1644.114909] Key type .llcrypt registered [ 1644.357382] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1644.376207] alg: No test for adler32 (adler32-zlib) [ 1645.245149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1645.260436] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 1652.370111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.142@tcp [ 1660.548265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1660.594420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1660.632283] LNet: Added LNI 192.168.203.42@tcp1 [8/256/0/180] [ 1661.925595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.42@tcp -d 192.168.203.142@tcp -m GET -e local_interrupt -r 2 [ 1661.950516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.42@tcp -d 192.168.203.142@tcp1 -m GET -e local_interrupt -r 2 [ 1661.977720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.42@tcp -d 192.168.203.42@tcp -m GET -e local_interrupt -r 2 [ 1662.014069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.42@tcp -d 192.168.203.42@tcp1 -m GET -e local_interrupt -r 2 [ 1662.039804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.42@tcp1 -d 192.168.203.142@tcp -m GET -e local_interrupt -r 2 [ 1662.068443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.42@tcp1 -d 192.168.203.142@tcp1 -m GET -e local_interrupt -r 2 [ 1662.100770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.42@tcp1 -d 192.168.203.42@tcp -m GET -e local_interrupt -r 2 [ 1662.127679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.42@tcp1 -d 192.168.203.42@tcp1 -m GET -e local_interrupt -r 2 [ 1662.156314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.42@tcp 192.168.203.142@tcp [ 1662.173922] LNet: There was an unexpected network error while writing to 192.168.203.142: rc = -22 [ 1662.178726] LNet: 1 local NIs in recovery (showing 1): 192.168.203.42@tcp [ 1662.371835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1662.396439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1662.419758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1662.446587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1662.472236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1662.501732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1662.526548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1662.574546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1662.591983] LNet: 88939:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1662.621849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1662.681950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1662.723362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1662.751483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1662.783451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1662.816919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1663.253611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.42@tcp -d 192.168.203.142@tcp -m GET -e local_dropped -r 2 [ 1663.290410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.42@tcp -d 192.168.203.142@tcp1 -m GET -e local_dropped -r 2 [ 1663.328154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.42@tcp -d 192.168.203.42@tcp -m GET -e local_dropped -r 2 [ 1663.369632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.42@tcp -d 192.168.203.42@tcp1 -m GET -e local_dropped -r 2 [ 1663.421742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.42@tcp1 -d 192.168.203.142@tcp -m GET -e local_dropped -r 2 [ 1663.477900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.42@tcp1 -d 192.168.203.142@tcp1 -m GET -e local_dropped -r 2 [ 1663.540154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.42@tcp1 -d 192.168.203.42@tcp -m GET -e local_dropped -r 2 [ 1663.587318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.42@tcp1 -d 192.168.203.42@tcp1 -m GET -e local_dropped -r 2 [ 1663.626328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.42@tcp 192.168.203.142@tcp [ 1663.647934] LNet: There was an unexpected network error while writing to 192.168.203.142: rc = -22 [ 1663.991762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1664.024417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1664.059875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1664.093421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1664.127902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1664.188437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1664.222315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1664.251238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1664.270323] LNet: 89059:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1664.299770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1664.344839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1664.377383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1664.410412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1664.444518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1664.477512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1664.950644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.42@tcp -d 192.168.203.142@tcp -m GET -e local_aborted -r 2 [ 1664.991355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.42@tcp -d 192.168.203.142@tcp1 -m GET -e local_aborted -r 2 [ 1665.019153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.42@tcp -d 192.168.203.42@tcp -m GET -e local_aborted -r 2 [ 1665.054307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.42@tcp -d 192.168.203.42@tcp1 -m GET -e local_aborted -r 2 [ 1665.091236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.42@tcp1 -d 192.168.203.142@tcp -m GET -e local_aborted -r 2 [ 1665.137023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.42@tcp1 -d 192.168.203.142@tcp1 -m GET -e local_aborted -r 2 [ 1665.181654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.42@tcp1 -d 192.168.203.42@tcp -m GET -e local_aborted -r 2 [ 1665.211718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.42@tcp1 -d 192.168.203.42@tcp1 -m GET -e local_aborted -r 2 [ 1665.246270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.42@tcp 192.168.203.142@tcp [ 1665.268648] LNet: There was an unexpected network error while writing to 192.168.203.142: rc = -22 [ 1665.499588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1665.534178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1665.587380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1665.622321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1665.687223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1665.732897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1665.780358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1665.807948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1665.837996] LNet: 89180:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1665.871525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1665.916519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1665.958277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1665.989534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1666.038316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1666.079606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1666.651314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.42@tcp -d 192.168.203.142@tcp -m GET -e local_no_route -r 2 [ 1666.711275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.42@tcp -d 192.168.203.142@tcp1 -m GET -e local_no_route -r 2 [ 1666.772441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.42@tcp -d 192.168.203.42@tcp -m GET -e local_no_route -r 2 [ 1666.816963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.42@tcp -d 192.168.203.42@tcp1 -m GET -e local_no_route -r 2 [ 1666.874349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.42@tcp1 -d 192.168.203.142@tcp -m GET -e local_no_route -r 2 [ 1666.915650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.42@tcp1 -d 192.168.203.142@tcp1 -m GET -e local_no_route -r 2 [ 1666.982150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.42@tcp1 -d 192.168.203.42@tcp -m GET -e local_no_route -r 2 [ 1667.022103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.42@tcp1 -d 192.168.203.42@tcp1 -m GET -e local_no_route -r 2 [ 1667.052885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.42@tcp 192.168.203.142@tcp [ 1667.256455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1667.282193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1667.320770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1667.351640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1667.399698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1667.446352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1667.482431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1667.509390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1667.535585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1667.558563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1667.580705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1667.600949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1667.630235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1667.651513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1668.161373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.42@tcp -d 192.168.203.142@tcp -m GET -e local_timeout -r 2 [ 1668.205215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.42@tcp -d 192.168.203.142@tcp1 -m GET -e local_timeout -r 2 [ 1668.234445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.42@tcp -d 192.168.203.42@tcp -m GET -e local_timeout -r 2 [ 1668.272272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.42@tcp -d 192.168.203.42@tcp1 -m GET -e local_timeout -r 2 [ 1668.302970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.42@tcp1 -d 192.168.203.142@tcp -m GET -e local_timeout -r 2 [ 1668.332165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.42@tcp1 -d 192.168.203.142@tcp1 -m GET -e local_timeout -r 2 [ 1668.358673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.42@tcp1 -d 192.168.203.42@tcp -m GET -e local_timeout -r 2 [ 1668.391300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.42@tcp1 -d 192.168.203.42@tcp1 -m GET -e local_timeout -r 2 [ 1668.422937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.42@tcp 192.168.203.142@tcp [ 1668.440801] LNet: There was an unexpected network error while writing to 192.168.203.142: rc = -22 [ 1668.444877] LNet: Skipped 1 previous similar message [ 1668.656864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1668.688318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1668.733981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1668.774959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1668.817426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1668.853976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1668.896238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1668.937441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1668.958714] LNet: 89420:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1668.968142] LNet: 89420:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1668.990660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1669.031557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1669.071681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1669.112877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1669.149478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1669.196713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1669.679752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.42@tcp -d 192.168.203.142@tcp -m GET -e remote_dropped -r 2 [ 1669.721742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.42@tcp -d 192.168.203.142@tcp1 -m GET -e remote_dropped -r 2 [ 1669.778476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.42@tcp -d 192.168.203.42@tcp -m GET -e remote_dropped -r 2 [ 1669.828459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.42@tcp -d 192.168.203.42@tcp1 -m GET -e remote_dropped -r 2 [ 1669.884611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.42@tcp1 -d 192.168.203.142@tcp -m GET -e remote_dropped -r 2 [ 1669.963302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.42@tcp1 -d 192.168.203.142@tcp1 -m GET -e remote_dropped -r 2 [ 1670.018462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.42@tcp1 -d 192.168.203.42@tcp -m GET -e remote_dropped -r 2 [ 1670.066723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.42@tcp1 -d 192.168.203.42@tcp1 -m GET -e remote_dropped -r 2 [ 1670.133432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.42@tcp 192.168.203.142@tcp [ 1670.170139] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.142@tcp [ 1670.384497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1670.411763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1670.433913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1670.455595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1670.478059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1670.508416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1670.530357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1670.549970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1670.580879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1670.618561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1670.655729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1670.693919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1670.746894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1670.782628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1676.257130] LNet: 89619:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1676.264250] LNetError: 89619:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1676.271564] LNet: Removed LNI 192.168.203.42@tcp [ 1676.282912] LNet: Removed LNI 192.168.203.42@tcp1 [ 1676.635655] Key type .llcrypt unregistered [ 1676.644026] Key type ._llcrypt unregistered [ 1690.621618] Key type ._llcrypt registered [ 1690.626252] Key type .llcrypt registered [ 1691.621085] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 15:45:52 (1787600752) [ 1692.244389] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1692.262479] alg: No test for adler32 (adler32-zlib) [ 1693.178663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1693.268198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1693.297544] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 1694.199701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1694.241584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.42@tcp [ 1694.271796] LNetError: 90598:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.42@tcp to recovery [ 1694.405879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.203.42@tcp [ 1694.576346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1700.273929] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 15:46:01 (1787600761) [ 1700.490932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1700.514523] LNet: 91187:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1700.519980] LNetError: 91187:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1700.536115] LNet: Removed LNI 192.168.203.42@tcp [ 1700.555128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1700.614580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1700.634049] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 1701.221222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1701.242166] LNet: Added LNI 192.168.203.42@tcp1 [8/256/0/180] [ 1702.039621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.203.42@tcp [ 1702.060900] LNetError: 91454:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.42@tcp to recovery [ 1702.624518] LNet: 1 local NIs in recovery (showing 1): 192.168.203.42@tcp [ 1702.627722] LNet: There was an unexpected network error while writing to 192.168.203.42: rc = -22 [ 1703.350746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.203.42@tcp [ 1711.519465] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 15:46:12 (1787600772) [ 1713.347930] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg342-server [ 1715.515698] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 15:46:15 (1787600775) [ 1717.547375] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg342-server [ 1718.946303] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 15:46:19 (1787600779) [ 1720.742076] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg342-server [ 1722.535876] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 15:46:23 (1787600783) [ 1722.637259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1722.661091] LNet: 93076:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1722.675132] LNetError: 93076:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1722.688100] LNet: Removed LNI 192.168.203.42@tcp [ 1722.707874] LNet: Removed LNI 192.168.203.42@tcp1 [ 1722.732948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1722.806628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1722.830971] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 1723.480565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1723.536647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1723.576599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1723.682258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1723.736906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1723.824442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1730.985141] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 15:46:31 (1787600791) [ 1731.076510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1731.104099] LNet: 93811:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1731.122436] LNetError: 93811:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1731.129627] LNetError: 93811:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1731.171514] LNet: Removed LNI 192.168.203.42@tcp [ 1731.226055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1731.279731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1731.323444] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 1739.300872] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 15:46:39 (1787600799) [ 1739.463559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1739.518184] LNet: 94512:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1739.535597] LNetError: 94512:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1739.566773] LNet: Removed LNI 192.168.203.42@tcp [ 1739.611780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1739.708613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1739.739452] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 1747.926919] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 15:46:48 (1787600808) [ 1749.355988] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1750.855524] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 15:46:51 (1787600811) [ 1751.398831] LNet: 95553:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1751.409634] LNetError: 95553:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1751.427853] LNet: Removed LNI 192.168.203.42@tcp [ 1752.168796] Key type .llcrypt unregistered [ 1752.177889] Key type ._llcrypt unregistered [ 1758.241243] Key type ._llcrypt registered [ 1758.244862] Key type .llcrypt registered [ 1758.522307] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1758.552764] alg: No test for adler32 (adler32-zlib) [ 1759.455630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1759.488440] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 1766.644450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.142@tcp [ 1771.120872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.203.142@tcp [ 1771.490291] LNet: 95979:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1771.496894] LNetError: 95979:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1771.513637] LNet: Removed LNI 192.168.203.42@tcp [ 1771.894858] Key type .llcrypt unregistered [ 1771.900676] Key type ._llcrypt unregistered [ 1782.615828] Key type ._llcrypt registered [ 1782.618834] Key type .llcrypt registered [ 1783.822829] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1785.239872] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1787.048162] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 15:47:27 (1787600847) [ 1788.493313] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1790.188417] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 15:47:31 (1787600851) [ 1791.763641] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg342-server [ 1793.089820] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 15:47:34 (1787600854) [ 1794.746517] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg342-server [ 1796.435862] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 15:47:37 (1787600857) [ 1797.743809] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1797.781915] alg: No test for adler32 (adler32-zlib) [ 1798.827408] Lustre: Lustre: Build Version: 2.17.57_80_g20f51fc [ 1799.075820] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 1800.839297] Key type lgssc registered [ 1802.189796] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1821.769585] Key type lgssc unregistered [ 1822.082341] LNet: 99154:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1822.088176] LNetError: 99154:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1822.101713] LNet: Removed LNI 192.168.203.42@tcp [ 1822.861703] Key type .llcrypt unregistered [ 1822.863971] Key type ._llcrypt unregistered [ 1837.022291] Key type ._llcrypt registered [ 1837.023785] Key type .llcrypt registered [ 1838.560826] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 15:48:18 (1787600898) [ 1838.978469] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1838.996697] alg: No test for adler32 (adler32-zlib) [ 1839.982531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1840.059476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1840.135527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1840.191188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1840.243498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1846.473184] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 15:48:27 (1787600907) [ 1846.573380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1846.593919] LNet: 100563:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1846.619271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1846.704458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1846.777503] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 1847.640698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1847.674428] LNet: 100714:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1847.679627] LNetError: 100714:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1847.714970] LNet: Removed LNI 192.168.203.42@tcp [ 1847.741461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1847.792315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1847.823090] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 1848.192568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1848.211862] LNetError: 100742:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1848.223298] LNet: Removed LNI 192.168.203.42@tcp [ 1848.245603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1848.349591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1848.389239] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 1854.584260] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 15:48:35 (1787600915) [ 1854.886891] LNet: 101333:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1854.894414] LNet: 101333:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1854.909113] LNetError: 101333:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1854.930325] LNet: Removed LNI 192.168.203.42@tcp [ 1855.705727] Key type .llcrypt unregistered [ 1855.709055] Key type ._llcrypt unregistered [ 1863.132797] Key type ._llcrypt registered [ 1863.136775] Key type .llcrypt registered [ 1863.298381] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1863.329691] alg: No test for adler32 (adler32-zlib) [ 1864.338772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1864.510541] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 1864.516611] LNet: *** cfs_fail_loc=e003, val=100*** [ 1865.619530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.42@tcp [ 1865.644156] LNet: *** cfs_fail_loc=e003, val=100*** [ 1865.646806] LNet: Skipped 2 previous similar messages [ 1865.654213] Ping reply received with mapping: [ 1865.656813] found metadata reply NI[192.168.203.42@tcp] [ 1865.668107] 2a cb a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1865.677451] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1865.684188] 5a 5a 5a 5a 5a 5a 5a 5a [ 1865.923540] LNet: 101735:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1865.931226] LNetError: 101735:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1865.944576] LNet: Removed LNI 192.168.203.42@tcp [ 1866.726738] Key type .llcrypt unregistered [ 1866.729805] Key type ._llcrypt unregistered [ 1879.494996] Key type ._llcrypt registered [ 1879.497819] Key type .llcrypt registered [ 1881.073304] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 15:49:01 (1787600941) [ 1882.086334] Key type .llcrypt unregistered [ 1882.092568] Key type ._llcrypt unregistered [ 1889.342953] Key type ._llcrypt registered [ 1889.346565] Key type .llcrypt registered [ 1889.730714] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1889.745652] alg: No test for adler32 (adler32-zlib) [ 1890.996178] LNetError: 102792:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1891.458908] Key type .llcrypt unregistered [ 1891.461729] Key type ._llcrypt unregistered [ 1897.486969] Key type ._llcrypt registered [ 1897.489544] Key type .llcrypt registered [ 1898.835874] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 15:49:19 (1787600959) [ 1899.531461] Key type .llcrypt unregistered [ 1899.535184] Key type ._llcrypt unregistered [ 1906.231443] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1906.239374] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1908.971270] Key type ._llcrypt registered [ 1908.973959] Key type .llcrypt registered [ 1909.249402] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1909.285854] alg: No test for adler32 (adler32-zlib) [ 1910.218449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1910.280060] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1920.968934] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 15:49:41 (1787600981) [ 1921.056793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1921.089551] LNet: 104576:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1921.093526] LNetError: 104576:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1921.122164] LNet: Removed LNI 10.1.2.3@tcp [ 1921.161094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1921.868271] Key type .llcrypt unregistered [ 1921.872175] Key type ._llcrypt unregistered [ 1927.570417] Key type ._llcrypt registered [ 1927.575305] Key type .llcrypt registered [ 1927.800433] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1927.834153] alg: No test for adler32 (adler32-zlib) [ 1928.783986] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 1928.828711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.42@tcp [ 1928.976068] LNet: 104873:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1928.981498] LNetError: 104873:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1929.015582] LNet: Removed LNI 192.168.203.42@tcp [ 1929.671821] Key type .llcrypt unregistered [ 1929.674575] Key type ._llcrypt unregistered [ 1940.310312] Key type ._llcrypt registered [ 1940.313180] Key type .llcrypt registered [ 1941.789205] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 15:50:02 (1787601002) [ 1942.375318] Key type .llcrypt unregistered [ 1942.377199] Key type ._llcrypt unregistered [ 1949.475735] Key type ._llcrypt registered [ 1949.484632] Key type .llcrypt registered [ 1949.670534] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1949.691958] alg: No test for adler32 (adler32-zlib) [ 1957.116267] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 15:50:18 (1787601018) [ 1957.477672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1957.541197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1957.630408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1957.681107] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 1958.352526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1958.400542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1958.443970] LNet: Removed LNI 192.168.203.42@tcp [ 1958.690310] LNet: 106628:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1958.990112] Key type .llcrypt unregistered [ 1958.992945] Key type ._llcrypt unregistered [ 1970.899792] Key type ._llcrypt registered [ 1970.901767] Key type .llcrypt registered [ 1972.240634] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 15:50:32 (1787601032) [ 1972.777113] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1972.790856] alg: No test for adler32 (adler32-zlib) [ 1973.698992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1973.760221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1973.788292] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 1974.437626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.42@tcp [ 1974.500813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.203.42@tcp [ 1974.523118] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1974.531404] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1974.536240] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1974, 0, 0 [ 1974.542126] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1974.544819] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1974.547718] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1974.550374] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1974, 0, 0 [ 1974.554259] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1974.556236] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1974.559262] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1974.562319] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1974, 0, 2024 [ 1974.567623] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1974.569796] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1974.571987] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1974.574232] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1974, 0, 2024 [ 1974.580468] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1981.402575] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 15:50:41 (1787601041) [ 1981.785787] LNet: 108167:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1981.791989] LNetError: 108167:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1981.807262] LNet: Removed LNI 192.168.203.42@tcp [ 1982.492128] Key type .llcrypt unregistered [ 1982.494433] Key type ._llcrypt unregistered [ 1988.633963] Key type ._llcrypt registered [ 1988.636687] Key type .llcrypt registered [ 1988.924773] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1988.952297] alg: No test for adler32 (adler32-zlib) [ 1989.872318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1989.893632] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 1997.064268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.142@tcp [ 2006.262590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2006.341244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2006.396105] LNet: Added LNI 192.168.203.42@tcp1 [8/256/0/180] [ 2007.480026] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2007.670809] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2007.678464] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2007.687775] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2010.178832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2010.206709] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2020.431984] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 15:51:21 (1787601081) [ 2020.552255] LNet: 109671:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2020.574297] LNetError: 109671:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2021.599206] LNet: Removed LNI 192.168.203.42@tcp [ 2021.613285] LNetError: 109671:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2021.633196] LNet: Removed LNI 192.168.203.42@tcp1 [ 2021.646333] LNet: Removed LNI 10.1.2.3@tcp99 [ 2022.494753] Key type .llcrypt unregistered [ 2022.496430] Key type ._llcrypt unregistered [ 2028.924531] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2028.939041] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2031.630381] Key type ._llcrypt registered [ 2031.635637] Key type .llcrypt registered [ 2031.863413] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2031.882194] alg: No test for adler32 (adler32-zlib) [ 2032.799650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2032.861676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2032.899480] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 2033.611093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2033.638765] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2035.667965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2035.719573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2035.759558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2035.782682] LNetError: 110428:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2035.822618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2035.873190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.42@tcp [ 2036.410831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2036.523160] LNet: 110448:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2036.539062] LNetError: 110448:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2036.552949] LNet: Removed LNI 10.1.2.3@tcp [ 2037.605759] LNet: Removed LNI 192.168.203.42@tcp [ 2038.496511] Key type .llcrypt unregistered [ 2038.498067] Key type ._llcrypt unregistered [ 2051.839421] Key type ._llcrypt registered [ 2051.841982] Key type .llcrypt registered [ 2053.177415] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 15:51:53 (1787601113) [ 2053.939623] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2053.979465] alg: No test for adler32 (adler32-zlib) [ 2054.939226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2055.041634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2055.074578] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 2055.889320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.42@tcp [ 2055.973717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg342-client@tcp [ 2063.817931] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 15:52:04 (1787601124) [ 2064.142593] LNet: 111986:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2064.149783] LNetError: 111986:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2064.176985] LNet: Removed LNI 192.168.203.42@tcp [ 2064.922412] Key type .llcrypt unregistered [ 2064.925521] Key type ._llcrypt unregistered [ 2072.199664] Key type ._llcrypt registered [ 2072.205852] Key type .llcrypt registered [ 2072.456791] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2072.488781] alg: No test for adler32 (adler32-zlib) [ 2073.450277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2073.490246] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 2073.527698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2080.246767] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 15:52:21 (1787601141) [ 2081.841189] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2083.755562] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 15:52:24 (1787601144) [ 2085.315765] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2087.100350] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 15:52:27 (1787601147) [ 2087.329468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2087.349757] LNet: 113412:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2087.354514] LNetError: 113412:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2087.374123] LNet: Removed LNI 192.168.203.42@tcp [ 2087.389716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2087.449677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2087.475775] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 2088.287664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2088.327793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2096.343646] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 15:52:36 (1787601156) [ 2096.555728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2096.581691] LNet: 114115:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2096.590377] LNetError: 114115:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2097.641967] LNet: Removed LNI 192.168.203.42@tcp [ 2097.670626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2097.748861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2097.781682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2098.185276] LNet: 114161:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2098.709358] Key type .llcrypt unregistered [ 2098.711312] Key type ._llcrypt unregistered [ 2111.722501] Key type ._llcrypt registered [ 2111.724907] Key type .llcrypt registered [ 2112.943162] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 15:52:53 (1787601173) [ 2113.326615] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2113.341962] alg: No test for adler32 (adler32-zlib) [ 2114.277101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2114.334588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2114.364220] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 2115.067811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2115.133678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.42@tcp [ 2121.533367] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 15:53:02 (1787601182) [ 2121.656523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2121.675846] LNet: 115677:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2121.683582] LNetError: 115677:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2121.700810] LNet: Removed LNI 192.168.203.42@tcp [ 2121.728861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2121.781898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2121.808531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2128.261099] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 15:53:09 (1787601189) [ 2128.483673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2128.509221] LNet: 116261:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2128.535927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2128.603265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2129.055392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2129.093722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2129.139404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2129.564950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2129.598937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2129.671237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2129.699817] LNet: 116294:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2129.746168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2129.820464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2130.341862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2130.378953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2130.476951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2131.019591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2131.056036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2136.983895] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 15:53:17 (1787601197) [ 2137.168460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2137.191427] LNet: 116893:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2137.230518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2137.296495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2137.336192] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 2138.060890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2138.076630] LNet: Added LNI 192.168.203.42@tcp2 [8/256/0/180] [ 2138.781248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2138.859393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.42@tcp [ 2139.097775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2139.116104] LNetError: 117173:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2139.121840] LNet: Removed LNI 192.168.203.42@tcp [ 2139.133809] LNet: Removed LNI 192.168.203.42@tcp2 [ 2139.158678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2139.233653] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 2139.238498] LNet: Added LNI 192.168.203.42@tcp2 [8/256/0/180] [ 2139.242784] LNet: 117179:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2139.748481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.42@tcp [ 2139.926901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2140.959613] LNet: Removed LNI 192.168.203.42@tcp [ 2140.963941] LNetError: 117217:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2140.975097] LNetError: 117217:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2141.001847] LNet: Removed LNI 192.168.203.42@tcp2 [ 2141.038714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2141.094207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2141.130749] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 2141.820106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2141.838383] LNet: Added LNI 192.168.203.42@tcp2 [8/256/0/180] [ 2142.565576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.42@tcp [ 2142.604246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2142.657445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.42@tcp [ 2142.832024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2142.849648] LNet: 117501:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2142.853571] LNet: 117501:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2143.903589] LNet: Removed LNI 192.168.203.42@tcp [ 2143.926523] LNetError: 117501:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2143.935744] LNetError: 117501:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2143.956178] LNet: Removed LNI 192.168.203.42@tcp2 [ 2143.987674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2144.062677] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 2144.074367] LNet: Added LNI 192.168.203.42@tcp2 [8/256/0/180] [ 2144.082285] LNet: 117507:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2144.679972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.42@tcp [ 2144.924281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2144.939911] LNet: Removed LNI 192.168.203.42@tcp [ 2145.964575] LNet: Removed LNI 192.168.203.42@tcp2 [ 2146.014928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2146.098251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2146.132262] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 2146.774848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2146.793158] LNet: Added LNI 192.168.203.42@tcp2 [8/256/0/180] [ 2147.629813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2147.724768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.42@tcp [ 2147.921179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2147.941941] LNetError: 117824:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2147.949727] LNetError: 117824:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2147.956304] LNet: Removed LNI 192.168.203.42@tcp [ 2147.965240] LNet: Removed LNI 192.168.203.42@tcp2 [ 2147.986285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2148.035046] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 2148.040640] LNet: Added LNI 192.168.203.42@tcp2 [8/256/0/180] [ 2148.044907] LNet: 117830:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2148.617103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.42@tcp [ 2148.776953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2149.855183] LNet: Removed LNI 192.168.203.42@tcp [ 2149.866714] LNet: Removed LNI 192.168.203.42@tcp2 [ 2149.910757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2149.976449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2150.006452] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 2150.665926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2150.699341] LNet: Added LNI 192.168.203.42@tcp2 [8/256/0/180] [ 2151.500649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.42@tcp [ 2151.547218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2151.601880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.42@tcp [ 2151.851583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2151.869789] LNet: 118151:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2151.874434] LNet: 118151:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2152.927369] LNet: Removed LNI 192.168.203.42@tcp [ 2153.959765] LNet: Removed LNI 192.168.203.42@tcp2 [ 2154.029400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2154.135346] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 2154.149888] LNet: Added LNI 192.168.203.42@tcp2 [8/256/0/180] [ 2154.164609] LNet: 118158:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2154.782565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.42@tcp [ 2155.036369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2155.066761] LNet: Removed LNI 192.168.203.42@tcp [ 2156.140986] LNet: Removed LNI 192.168.203.42@tcp2 [ 2156.233382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2156.346889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2156.389444] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 2157.203804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2157.230884] LNet: Added LNI 192.168.203.42@tcp2 [8/256/0/180] [ 2158.333365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2158.385230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2158.456332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.42@tcp [ 2158.889635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2158.911372] LNetError: 118491:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2158.915263] LNetError: 118491:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2158.927641] LNet: Removed LNI 192.168.203.42@tcp [ 2158.936462] LNet: Removed LNI 192.168.203.42@tcp2 [ 2158.970337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2159.089102] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 2159.114866] LNet: Added LNI 192.168.203.42@tcp2 [8/256/0/180] [ 2159.126920] LNet: 118497:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2159.820771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.42@tcp [ 2160.343871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2161.439171] LNet: Removed LNI 192.168.203.42@tcp [ 2161.473462] LNet: Removed LNI 192.168.203.42@tcp2 [ 2161.503920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2161.577332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2161.609109] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 2162.458891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2162.476648] LNet: Added LNI 192.168.203.42@tcp2 [8/256/0/180] [ 2163.182603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.42@tcp [ 2163.234327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2163.282824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2163.360465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.42@tcp [ 2163.890770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2164.959173] LNet: Removed LNI 192.168.203.42@tcp [ 2165.992865] LNet: Removed LNI 192.168.203.42@tcp2 [ 2166.037854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2166.108747] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 2166.118336] LNet: Added LNI 192.168.203.42@tcp2 [8/256/0/180] [ 2166.844248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.42@tcp [ 2175.299156] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2177.434742] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 15:53:57 (1787601237) [ 2177.813322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2177.837894] LNet: 119617:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2177.850233] LNet: 119617:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2177.866035] LNetError: 119617:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2177.874989] LNetError: 119617:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2177.885160] LNet: Removed LNI 192.168.203.42@tcp [ 2178.986685] LNet: Removed LNI 192.168.203.42@tcp2 [ 2179.063191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2179.637923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2179.688419] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 2180.830458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2180.858896] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2182.298745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2182.365495] LNet: Removed LNI 10.1.2.3@tcp [ 2182.380924] LNet: Removed LNI 192.168.203.42@tcp [ 2182.419673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2182.524060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2182.577557] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 2182.593052] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2183.255150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2183.299519] LNet: Removed LNI 192.168.203.42@tcp [ 2183.311362] LNet: Removed LNI 10.1.2.3@tcp [ 2190.884914] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 15:54:11 (1787601251) [ 2190.948489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2191.014754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2191.071832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2191.110288] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 2192.016833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2192.078449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.42@tcp [ 2192.133091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.42@tcp [ 2192.183440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.42@tcp [ 2192.225961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.42@tcp [ 2192.272825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.42@tcp [ 2192.308717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.42@tcp [ 2192.333142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.42@tcp [ 2192.364903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.42@tcp [ 2192.397572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.42@tcp [ 2192.442330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.42@tcp [ 2192.607440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2199.005633] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 15:54:19 (1787601259) [ 2199.092293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2200.171056] LNet: Removed LNI 192.168.203.42@tcp [ 2200.216210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2200.314639] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2200.384381] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2200.389503] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2200.397982] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2202.781904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2202.805253] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 2203.480267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2203.498136] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2205.375332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.42@tcp [ 2205.411901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2205.477278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.203.42@tcp -m PUT -e local_timeout -r 1 [ 2205.527733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2205.597449] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2205.604940] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2205.963634] LNetError: 121297:0:(lib-move.c:4801:lnet_parse()) 192.168.203.42@tcp, src 192.168.203.42@tcp: Dropping PUT (error -108 looking up sender) [ 2207.010615] LNet: Removed LNI 10.1.2.3@tcp [ 2208.040069] LNet: Removed LNI 192.168.203.42@tcp [ 2208.588837] Key type .llcrypt unregistered [ 2208.592071] Key type ._llcrypt unregistered [ 2219.557352] Key type ._llcrypt registered [ 2219.559164] Key type .llcrypt registered [ 2220.765286] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 15:54:41 (1787601281) [ 2221.124568] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2221.147676] alg: No test for adler32 (adler32-zlib) [ 2222.014826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2222.325523] LNet: 122612:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2223.232585] Key type .llcrypt unregistered [ 2223.234356] Key type ._llcrypt unregistered [ 2232.128746] Key type ._llcrypt registered [ 2232.133197] Key type .llcrypt registered [ 2232.461566] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2232.487246] alg: No test for adler32 (adler32-zlib) [ 2233.537985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2233.583525] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 2242.246786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.142@tcp [ 2242.536364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2243.425759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.142@tcp [ 2243.842752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2243.900652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2243.936140] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.142@tcp [ 2243.942252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2243.986552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2244.022225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2244.080510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2244.166985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2244.250622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2244.291711] LNet: 123067:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2244.325233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2244.368465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2244.427107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2244.470396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2244.512869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2244.556533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2245.477493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2245.748751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2245.777567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2245.806727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2245.838698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2245.866979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2245.918187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2245.948515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2245.983960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2246.008261] LNet: 123184:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2246.026316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2246.060422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2246.097414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2246.122682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2246.167595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2246.221137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2246.780876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.142@tcp [ 2247.155013] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2247.191290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2247.249703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2247.298947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2247.341355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2247.387024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2247.426201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2247.474734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2247.496506] LNet: 123302:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2247.527657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2247.580592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2247.636708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2247.675764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2247.727987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2247.775876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2248.594105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2248.869307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2248.926525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2248.968091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2249.037259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2249.073374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2249.110566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2249.176670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2249.219400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2249.258798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2249.301430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2249.341624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2249.375848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2249.415437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2249.447577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2249.834904] LNet: 123469:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2249.842465] LNetError: 123469:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2249.869035] LNet: Removed LNI 192.168.203.42@tcp [ 2250.200835] Key type .llcrypt unregistered [ 2250.203772] Key type ._llcrypt unregistered [ 2269.573041] Key type ._llcrypt registered [ 2269.578926] Key type .llcrypt registered [ 2271.446543] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 15:55:31 (1787601331) [ 2272.003803] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2272.048531] alg: No test for adler32 (adler32-zlib) [ 2273.074907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2273.509452] LNet: 124355:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2274.276369] Key type .llcrypt unregistered [ 2274.278101] Key type ._llcrypt unregistered [ 2281.217305] Key type ._llcrypt registered [ 2281.220538] Key type .llcrypt registered [ 2281.398547] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2281.407509] alg: No test for adler32 (adler32-zlib) [ 2282.321562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2282.361606] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 2290.255409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.142@tcp [ 2300.311208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2300.358766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2300.387311] LNet: Added LNI 192.168.203.42@tcp1 [8/256/0/180] [ 2301.394637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2302.035903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.142@tcp [ 2302.330781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2302.361834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2302.408181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2302.436625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2302.477772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2302.523433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2302.556795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2302.591988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2302.616926] LNet: 125011:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2302.647866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2302.692680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2302.726942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2302.792740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2302.841336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2302.890676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2303.552975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2303.814272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2303.855629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2303.892599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2303.905588] LNet: 2 peer NIs in recovery (showing 2): 192.168.203.142@tcp, 192.168.203.142@tcp1 [ 2303.927785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2303.956789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2303.988921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2304.043592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2304.084947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2304.138288] LNet: 125128:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2304.165539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2304.231215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2304.291994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2304.337389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2304.367611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2304.413733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2305.171351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.142@tcp [ 2305.392883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2305.421807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2305.445204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2305.488699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2305.532975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2305.574812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2305.605595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2305.636290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2305.650376] LNet: 125246:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2305.663603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2305.684887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2305.708904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2305.736139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2305.759968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2305.788630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2306.334518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2306.623748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2306.664882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2306.707433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2306.749846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2306.792484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2306.847648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2306.930318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2306.983547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2307.027639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2307.063473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2307.094352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2307.138950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2307.198853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2307.251304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2312.398839] LNet: 125448:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2312.414650] LNetError: 125448:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2312.427480] LNet: Removed LNI 192.168.203.42@tcp [ 2312.450644] LNet: Removed LNI 192.168.203.42@tcp1 [ 2312.941931] Key type .llcrypt unregistered [ 2312.949230] Key type ._llcrypt unregistered [ 2330.185232] Key type ._llcrypt registered [ 2330.188800] Key type .llcrypt registered [ 2331.718265] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 15:56:32 (1787601392) [ 2332.232731] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2332.246986] alg: No test for adler32 (adler32-zlib) [ 2333.178215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2333.237850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2333.258253] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 2333.948474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2333.983556] LNet: Added LNI 192.168.203.42@tcp2 [8/256/0/180] [ 2334.846429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.42@tcp [ 2334.901961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2335.967314] LNet: Removed LNI 192.168.203.42@tcp [ 2342.516859] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 15:56:43 (1787601403) [ 2344.054659] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2346.278737] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 15:56:46 (1787601406) [ 2354.144800] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 15:56:54 (1787601414) [ 2361.768897] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 15:57:02 (1787601422) [ 2370.827696] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 15:57:10 (1787601430) [ 2379.174736] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 15:57:19 (1787601439) [ 2386.246913] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 15:57:26 (1787601446) [ 2394.120203] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 15:57:34 (1787601454) [ 2402.730458] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 15:57:43 (1787601463) [ 2412.079750] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 15:57:52 (1787601472) [ 2419.106261] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 15:57:59 (1787601479) [ 2426.150724] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 15:58:06 (1787601486) [ 2433.420502] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 15:58:14 (1787601494) [ 2440.457186] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 15:58:21 (1787601501) [ 2446.479993] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 15:58:27 (1787601507) [ 2455.283573] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 15:58:35 (1787601515) [ 2465.770820] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 15:58:45 (1787601525) [ 2474.403840] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 15:58:54 (1787601534) [ 2483.389054] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 15:59:03 (1787601543) [ 2483.454439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2483.490079] LNet: 137143:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2483.496958] LNetError: 137143:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2484.539107] LNet: Removed LNI 192.168.203.42@tcp2 [ 2484.576815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2484.750332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2484.789986] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 2485.809978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2485.829066] LNet: 137294:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2485.841909] LNetError: 137294:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2485.856421] LNet: Removed LNI 192.168.203.42@tcp [ 2485.894168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2485.960311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2485.984024] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 2494.435989] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 15:59:15 (1787601555) [ 2501.435677] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 15:59:22 (1787601562) [ 2508.093837] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 15:59:28 (1787601568) [ 2515.666992] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 15:59:36 (1787601576) [ 2523.254953] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 15:59:43 (1787601583) [ 2531.031106] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 15:59:51 (1787601591) [ 2539.182684] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 15:59:59 (1787601599) [ 2546.980298] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 16:00:07 (1787601607) [ 2554.122267] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 16:00:14 (1787601614) [ 2561.001426] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 16:00:21 (1787601621) [ 2568.248248] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 16:00:28 (1787601628) [ 2576.026824] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 16:00:36 (1787601636) [ 2584.963621] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2316 sec ========== 16:00:45 (1787601645) [ 2585.305213] LNet: 144760:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2585.316907] LNetError: 144760:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2585.331157] LNet: Removed LNI 192.168.203.42@tcp [ 2586.689946] Key type .llcrypt unregistered [ 2586.696829] Key type ._llcrypt unregistered [ 2602.933752] Key type ._llcrypt registered [ 2602.943147] Key type .llcrypt registered [ 2603.482751] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2603.514279] alg: No test for adler32 (adler32-zlib) [ 2604.783173] Lustre: Lustre: Build Version: 2.17.57_80_g20f51fc [ 2605.170629] LNet: Added LNI 192.168.203.42@tcp [8/256/0/180] [ 2606.863595] Key type lgssc registered [ 2608.171242] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2667.832039] Lustre: Mounted lustre-client - version 2.17.57_80_g20f51fc [ 2672.738543] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2679.823047] Lustre: Unmounted lustre-client [ 2700.383591] Key type lgssc unregistered [ 2700.719795] LNet: 147733:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2700.728333] LNetError: 147733:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2700.772708] LNet: Removed LNI 192.168.203.42@tcp [ 2701.630275] Key type .llcrypt unregistered [ 2701.636342] Key type ._llcrypt unregistered