[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 444083718 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524588K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002391] x2apic enabled [ 0.003010] Switched APIC routing to physical x2apic. [ 0.004016] kvm-guest: setup PV IPIs [ 0.007691] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009014] pid_max: default: 32768 minimum: 301 [ 0.010139] LSM: Security Framework initializing [ 0.011054] Yama: becoming mindful. [ 0.012040] SELinux: Initializing. [ 0.013091] *** VALIDATE selinux *** [ 0.021685] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026010] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027171] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028115] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029116] *** VALIDATE tmpfs *** [ 0.031261] *** VALIDATE proc *** [ 0.032289] *** VALIDATE cgroup *** [ 0.033012] *** VALIDATE cgroup2 *** [ 0.034299] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035167] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037033] Spectre V2 : User space: Vulnerable [ 0.038010] Speculative Store Bypass: Vulnerable [ 0.040692] debug: unmapping init [mem 0xffffffffb9659000-0xffffffffb9660fff] [ 0.042185] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043767] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044063] ... version: 2 [ 0.045012] ... bit width: 48 [ 0.046013] ... generic registers: 4 [ 0.047012] ... value mask: 0000ffffffffffff [ 0.048018] ... max period: 00007fffffffffff [ 0.049017] ... fixed-purpose events: 3 [ 0.050015] ... event mask: 000000070000000f [ 0.051348] rcu: Hierarchical SRCU implementation. [ 0.053509] smp: Bringing up secondary CPUs ... [ 0.054526] x86: Booting SMP configuration: [ 0.055025] .... node #0, CPUs: #1 #2 #3 [ 0.062018] smp: Brought up 1 node, 4 CPUs [ 0.064014] smpboot: Max logical packages: 1 [ 0.065020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.142222] node 0 deferred pages initialised in 74ms [ 0.145478] devtmpfs: initialized [ 0.147284] x86/mm: Memory block size: 128MB [ 0.150041] gcov: version magic: 0x41383552 [ 0.154014] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.155082] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.156320] pinctrl core: initialized pinctrl subsystem [ 0.157201] [ 0.157783] ************************************************************* [ 0.158012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.159032] ** ** [ 0.160016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.161014] ** ** [ 0.162015] ** This means that this kernel is built to expose internal ** [ 0.163014] ** IOMMU data structures, which may compromise security on ** [ 0.164015] ** your system. ** [ 0.165013] ** ** [ 0.166012] ** If you see this message and you are not debugging the ** [ 0.167015] ** kernel, report this immediately to your vendor! ** [ 0.168024] ** ** [ 0.169012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.170014] ************************************************************* [ 0.171759] NET: Registered protocol family 16 [ 0.172528] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.173063] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.174070] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.175460] cpuidle: using governor menu [ 0.177859] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.180529] PCI: Using configuration type 1 for base access [ 0.182109] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.191056] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.192036] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.194078] cryptd: max_cpu_qlen set to 1000 [ 0.196115] ACPI: Added _OSI(Module Device) [ 0.197013] ACPI: Added _OSI(Processor Device) [ 0.198010] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.199014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.203522] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.205614] ACPI: Interpreter enabled [ 0.206059] ACPI: PM: (supports S0 S3 S4 S5) [ 0.207012] ACPI: Using IOAPIC for interrupt routing [ 0.208098] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.209387] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.218643] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.219048] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.220026] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.221080] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.223398] acpiphp: Slot [2] registered [ 0.224176] acpiphp: Slot [5] registered [ 0.225173] acpiphp: Slot [6] registered [ 0.226148] acpiphp: Slot [3] registered [ 0.227117] acpiphp: Slot [4] registered [ 0.228121] acpiphp: Slot [7] registered [ 0.229164] acpiphp: Slot [8] registered [ 0.230196] acpiphp: Slot [9] registered [ 0.231114] acpiphp: Slot [10] registered [ 0.232119] acpiphp: Slot [11] registered [ 0.233169] acpiphp: Slot [12] registered [ 0.234142] acpiphp: Slot [13] registered [ 0.235136] acpiphp: Slot [14] registered [ 0.236120] acpiphp: Slot [15] registered [ 0.237130] acpiphp: Slot [16] registered [ 0.238120] acpiphp: Slot [17] registered [ 0.239112] acpiphp: Slot [18] registered [ 0.240123] acpiphp: Slot [19] registered [ 0.241158] acpiphp: Slot [20] registered [ 0.242150] acpiphp: Slot [21] registered [ 0.243129] acpiphp: Slot [22] registered [ 0.245133] acpiphp: Slot [23] registered [ 0.247122] acpiphp: Slot [24] registered [ 0.249143] acpiphp: Slot [25] registered [ 0.250147] acpiphp: Slot [26] registered [ 0.252122] acpiphp: Slot [27] registered [ 0.253110] acpiphp: Slot [28] registered [ 0.255116] acpiphp: Slot [29] registered [ 0.257105] acpiphp: Slot [30] registered [ 0.258110] acpiphp: Slot [31] registered [ 0.260099] PCI host bridge to bus 0000:00 [ 0.262021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.264023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.267035] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.270041] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.272027] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.275025] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.276231] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.280077] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.283360] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.290569] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.295037] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.298020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.301029] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.303016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.307037] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.309926] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.313047] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.314901] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.320013] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.331018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.335016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.340186] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.345011] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.350012] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.363018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.372322] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.379018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.387015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.412016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.424293] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.427424] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.430375] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.432393] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.436243] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.440027] iommu: Default domain type: Passthrough [ 0.442393] SCSI subsystem initialized [ 0.444149] ACPI: bus type USB registered [ 0.445111] usbcore: registered new interface driver usbfs [ 0.448085] usbcore: registered new interface driver hub [ 0.450083] usbcore: registered new device driver usb [ 0.452177] pps_core: LinuxPPS API ver. 1 registered [ 0.455015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.458062] PTP clock support registered [ 0.461052] EDAC MC: Ver: 3.0.0 [ 0.463117] PCI: Using ACPI for IRQ routing [ 0.464814] NetLabel: Initializing [ 0.467012] NetLabel: domain hash size = 128 [ 0.469010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.471082] NetLabel: unlabeled traffic allowed by default [ 0.473067] vgaarb: loaded [ 0.475280] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.476011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.485198] clocksource: Switched to clocksource kvm-clock [ 0.588855] VFS: Disk quotas dquot_6.6.0 [ 0.590849] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.594116] *** VALIDATE ramfs *** [ 0.595577] *** VALIDATE hugetlbfs *** [ 0.597340] pnp: PnP ACPI init [ 0.599813] pnp: PnP ACPI: found 6 devices [ 0.624976] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.628920] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.631622] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.633657] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.636143] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.638900] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.642271] NET: Registered protocol family 2 [ 0.644756] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.650880] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.655472] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.660294] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.664094] TCP: Hash tables configured (established 65536 bind 65536) [ 0.667283] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.670860] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.674352] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.677327] NET: Registered protocol family 1 [ 0.681139] RPC: Registered named UNIX socket transport module. [ 0.683375] RPC: Registered udp transport module. [ 0.685282] RPC: Registered tcp transport module. [ 0.687057] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.689842] NET: Registered protocol family 44 [ 0.690952] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.693100] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.695084] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.697576] PCI: CLS 0 bytes, default 64 [ 0.699738] Unpacking initramfs... [ 2.278724] debug: unmapping init [mem 0xffff9af7fcc64000-0xffff9af7fffcffff] [ 2.282821] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.285114] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.287899] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.789867] Initialise system trusted keyrings [ 2.791687] Key type blacklist registered [ 2.793546] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.802753] zbud: loaded [ 2.806221] *** VALIDATE nfs *** [ 2.807413] *** VALIDATE nfs4 *** [ 2.808829] pstore: using deflate compression [ 2.813644] Platform Keyring initialized [ 2.919462] NET: Registered protocol family 38 [ 2.921731] Key type asymmetric registered [ 2.924354] Asymmetric key parser 'x509' registered [ 2.926343] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.928758] io scheduler mq-deadline registered [ 2.930587] io scheduler kyber registered [ 2.932424] io scheduler bfq registered [ 2.934393] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.937625] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.939669] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.942309] ACPI: Power Button [PWRF] [ 2.948418] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.955330] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.969339] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.999377] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.026434] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.032448] Non-volatile memory driver v1.3 [ 3.034288] Linux agpgart interface v0.103 [ 3.090685] virtio_blk virtio1: [vda] 150400 512-byte logical blocks (77.0 MB/73.4 MiB) [ 3.094656] vda: detected capacity change from 0 to 77004800 [ 3.128331] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.130764] vdb: detected capacity change from 0 to 1073741824 [ 3.137335] libphy: Fixed MDIO Bus: probed [ 3.145253] usbcore: registered new interface driver usbserial_generic [ 3.147391] usbserial: USB Serial support registered for generic [ 3.149619] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.155199] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.156970] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.158963] mousedev: PS/2 mouse device common for all mice [ 3.162487] rtc_cmos 00:05: RTC can wake from S4 [ 3.166183] rtc_cmos 00:05: registered as rtc0 [ 3.168533] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.168749] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.170708] intel_pstate: CPU model not supported [ 3.175579] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.179587] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.181137] hid: raw HID events driver (C) Jiri Kosina [ 3.184624] usbcore: registered new interface driver usbhid [ 3.186662] usbhid: USB HID core driver [ 3.188257] drop_monitor: Initializing network drop monitor service [ 3.190527] Initializing XFRM netlink socket [ 3.192682] NET: Registered protocol family 10 [ 3.195467] Segment Routing with IPv6 [ 3.196855] NET: Registered protocol family 17 [ 3.199618] mpls_gso: MPLS GSO support [ 3.204442] RAS: Correctable Errors collector initialized. [ 3.206235] AVX version of gcm_enc/dec engaged. [ 3.207446] AES CTR mode by8 optimization enabled [ 3.281905] sched_clock: Marking stable (3281855876, 0)->(4330667071, -1048811195) [ 3.285446] registered taskstats version 1 [ 3.288183] Loading compiled-in X.509 certificates [ 3.290669] zswap: loaded using pool lzo/zbud [ 3.315553] Key type big_key registered [ 3.325974] Key type encrypted registered [ 3.327603] ima: No TPM chip found, activating TPM-bypass! [ 3.329223] ima: Allocated hash algorithm: sha1 [ 3.330806] ima: No architecture policies found [ 3.332491] evm: Initialising EVM extended attributes: [ 3.334241] evm: security.selinux [ 3.335483] evm: security.ima [ 3.336571] evm: security.capability [ 3.337939] evm: HMAC attrs: 0x1 [ 3.340212] rtc_cmos 00:05: setting system clock to 2026-10-01 22:53:40 UTC (1790895220) [ 3.346254] debug: unmapping init [mem 0xffffffffba603000-0xffffffffba7fffff] [ 3.349313] debug: unmapping init [mem 0xffffffffb9382000-0xffffffffb9658fff] [ 3.359123] Write protecting the kernel read-only data: 28672k [ 3.362736] debug: unmapping init [mem 0xffffffffb7a03000-0xffffffffb7bfffff] [ 3.365496] debug: unmapping init [mem 0xffffffffb8314000-0xffffffffb83fffff] [ 3.399458] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.407445] systemd[1]: Detected virtualization kvm. [ 3.408719] systemd[1]: Detected architecture x86-64. [ 3.410052] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.439280] systemd[1]: No hostname configured. [ 3.440563] systemd[1]: Set hostname to . [ 3.442014] random: systemd: uninitialized urandom read (16 bytes read) [ 3.443852] systemd[1]: Initializing machine ID from random generator. [ 3.490811] random: ln: uninitialized urandom read (6 bytes read) [ 3.560604] random: systemd: uninitialized urandom read (16 bytes read) [ 3.564096] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.569186] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.574255] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Reached target Timers. [ OK ] Reached target Local File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... [ OK ] Reached target Sockets. Starting Setup Virtual Console... Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Reached target Swap. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.177727] device-mapper: uevent: version 1.0.3 [ 4.180115] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ 4.867580] random: fast init done [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.956092] virtio_net virtio0 ens2: renamed from eth0 [ 4.998260] scsi host0: ata_piix [ 5.019906] scsi host1: ata_piix [ 5.024208] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.026975] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.774727] dracut-initqueue[576]: RTNETLINK answers: File exists [ 9.843473] random: crng init done [ 9.847438] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.299563] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.548586] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.825236] SELinux: Disabled at runtime. [ 11.895055] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.904612] systemd[1]: Detected virtualization kvm. [ 11.907108] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.422645] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.426567] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.432564] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.438320] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.442213] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.450878] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.457216] systemd[1]: Reached target rpc_pipefs.target. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Switch Root. [ OK ] Created slice system-getty.slice. Mounting POSIX Message Queue File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Remount Root and Kernel File Systems... Starting Create list of required st…ce nodes for the current kernel... Mounting Huge Pages File System... [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target RPC Port Mapper. Mounting Kernel Debug File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ 12.705861] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 13.011138] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.328957] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.369879] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.545854] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.557173] EDAC sbridge: Ver: 1.1.2 [ 14.898380] Key type dns_resolver registered [ 15.212731] NFS: Registering the id_resolver key type [ 15.215057] Key type id_resolver registered [ 15.216640] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg403-client login: [ 51.855910] libcfs: loading out-of-tree module taints kernel. [ 52.090836] Key type ._llcrypt registered [ 52.111257] Key type .llcrypt registered [ 52.919658] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 52.959941] alg: No test for adler32 (adler32-zlib) [ 54.714471] Lustre: Lustre: Build Version: 2.17.58_63_g3705aed [ 55.920684] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 57.834332] Key type lgssc registered [ 60.136525] Lustre: Echo OBD driver; http://www.lustre.org/ [ 202.071537] Lustre: Mounted lustre-client - version 2.17.58_63_g3705aed [ 207.329791] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 221.251635] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing check_logdir /tmp/testlogs/ [ 226.036666] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing yml_node [ 227.815200] Lustre: lustre-OST0000-osc-ffff9af846ae6000: disconnect after 23s idle [ 231.211672] Lustre: DEBUG MARKER: Client: 2.17.58.63 [ 234.480940] Lustre: DEBUG MARKER: MDS: 2.17.58.63 [ 238.560667] Lustre: DEBUG MARKER: OSS: 2.17.58.63 [ 240.451722] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Thu Oct 1 18:57:35 EDT 2026 [ 257.629485] Lustre: Unmounted lustre-client [ 279.331643] Key type lgssc unregistered [ 279.602041] LNet: 8405:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 279.621096] LNetError: 8405:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 279.652603] LNet: Removed LNI 192.168.204.3@tcp [ 280.461174] Key type .llcrypt unregistered [ 280.465394] Key type ._llcrypt unregistered [ 305.824478] Key type ._llcrypt registered [ 305.832042] Key type .llcrypt registered [ 306.132519] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 306.157356] alg: No test for adler32 (adler32-zlib) [ 307.086481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 307.170636] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 307.212118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 307.945302] hrtimer: interrupt took 14282150 ns [ 312.190431] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 312.291408] LNet: 9287:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 312.296625] LNetError: 9287:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 312.309053] LNet: Removed LNI 192.168.204.3@tcp [ 313.059451] Key type .llcrypt unregistered [ 313.066654] Key type ._llcrypt unregistered [ 321.569336] Key type ._llcrypt registered [ 321.571984] Key type .llcrypt registered [ 323.399780] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 18:58:58 (1790895538) [ 323.756284] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 323.768475] alg: No test for adler32 (adler32-zlib) [ 324.635349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 324.749496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 324.810457] LNet: 9706:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 324.850451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 324.939498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 324.954711] LNet: 9713:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 333.406794] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 18:59:08 (1790895548) [ 333.716747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 333.740954] LNet: 10291:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 333.758845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 333.849050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 340.941643] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 18:59:16 (1790895556) [ 341.234408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 341.282876] LNet: 10877:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 341.303947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 341.427548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 347.874743] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 18:59:23 (1790895563) [ 348.130053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 348.167752] LNet: 11461:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 348.193698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 348.285713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 355.637829] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 18:59:31 (1790895571) [ 355.800683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 355.845746] LNet: 12041:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 355.930837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 356.053209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 356.871440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 365.249387] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 18:59:40 (1790895580) [ 365.548325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 365.581870] LNet: 12637:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 365.644649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 365.815763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 366.576757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 366.936738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 377.052370] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 18:59:51 (1790895591) [ 377.315429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 377.433833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 377.638580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 387.915553] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 19:00:02 (1790895602) [ 388.152600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 388.200343] LNet: 13817:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 388.212261] LNet: 13817:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 388.287497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 388.511623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 388.576362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 389.375915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 389.416878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 390.222371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 390.306449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 390.743832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 390.794508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 391.463825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 391.540920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 392.472794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 392.609325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 393.617359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 393.692358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 394.219485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 394.280103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 394.899361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 394.937277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 395.301592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 395.336413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 395.708893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 395.743242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 396.480330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 396.533118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 397.145474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 397.215516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 404.190879] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 19:00:20 (1790895620) [ 404.484490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 404.581538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 404.700566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 404.796065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 413.089197] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 19:00:28 (1790895628) [ 413.217652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 413.278957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 413.411193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 413.470901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 420.609540] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 19:00:36 (1790895636) [ 420.772389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 420.814134] LNet: 15673:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 420.822673] LNet: 15673:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 420.874617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 421.006631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 421.094733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 429.676354] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 19:00:45 (1790895645) [ 429.943744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 430.019724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 430.103408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 430.135776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 437.388894] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 19:00:52 (1790895652) [ 437.735413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 437.811100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 437.932464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 437.992545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 445.761196] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 19:01:01 (1790895661) [ 445.885327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 445.936218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 446.086390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 446.124731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 455.086805] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 19:01:10 (1790895670) [ 455.326735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 455.378758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 455.500637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 456.202363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 464.469841] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 19:01:20 (1790895680) [ 464.550661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 464.610527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 464.817438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 465.087353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 474.396477] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 19:01:29 (1790895689) [ 474.552277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 474.640406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 474.840884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 475.255077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 483.268762] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 19:01:38 (1790895698) [ 483.435150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 483.544470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 484.331738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 484.821302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 493.785220] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 19:01:49 (1790895709) [ 494.173125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 494.232092] LNet: 20523:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 494.238695] LNet: 20523:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 494.278829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 494.374387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 494.435656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 501.883436] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 19:01:57 (1790895717) [ 502.069947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 502.155859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 502.287036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 502.354176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 510.674563] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 19:02:06 (1790895726) [ 510.744753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 510.802640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 510.953378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 518.783909] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 19:02:14 (1790895734) [ 519.029870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 519.166909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 519.305356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 527.455194] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 19:02:22 (1790895742) [ 527.719225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 527.756843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 527.903411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 528.470506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 535.754505] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 19:02:31 (1790895751) [ 535.888744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 535.930387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 536.062931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 536.100668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 543.539635] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 19:02:39 (1790895759) [ 543.887818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 543.935660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 544.032922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 544.067098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 551.942717] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 19:02:47 (1790895767) [ 552.042891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 552.082895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 552.146464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 552.173233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 559.427943] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 19:02:55 (1790895775) [ 559.684962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 559.728194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 559.794773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 559.838069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 567.468695] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 19:03:02 (1790895782) [ 567.619615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 567.731201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 567.830950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 567.970307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 568.011326] LNetError: 25840:0:(peer.c:2366:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 568.088243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 568.155325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 577.044729] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 19:03:12 (1790895792) [ 577.213944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 577.333947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 577.479435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.BhaUIBFaSG [ 577.644939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.BhaUIBFaSG [ 577.758846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.BhaUIBFaSG [ 586.665626] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 19:03:22 (1790895802) [ 586.976924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 587.053337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 587.135151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 587.187616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 587.235639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 594.767274] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 19:03:30 (1790895810) [ 594.934911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 595.012795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 595.090405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 595.122897] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 596.651265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.3@tcp [ 598.123518] Lustre: Lustre: Build Version: 2.17.58_63_g3705aed [ 600.088258] Key type lgssc registered [ 601.477762] Lustre: Echo OBD driver; http://www.lustre.org/ [ 605.652508] LNetError: 27603:0:(lib-move.c:2151:lnet_handle_find_routed_path()) peer 192.168.204.3@tcp99 has no available nets [ 608.367290] Key type lgssc unregistered [ 608.780837] LNetError: 28718:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 608.823904] LNet: Removed LNI 192.168.204.3@tcp [ 609.415221] Key type .llcrypt unregistered [ 609.420400] Key type ._llcrypt unregistered [ 616.735609] Key type ._llcrypt registered [ 616.738849] Key type .llcrypt registered [ 617.956204] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 19:03:53 (1790895833) [ 618.437537] Key type .llcrypt unregistered [ 618.441298] Key type ._llcrypt unregistered [ 624.345193] Key type ._llcrypt registered [ 624.347873] Key type .llcrypt registered [ 624.563331] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 624.577177] alg: No test for adler32 (adler32-zlib) [ 625.482682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 625.540800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 625.565873] LNet: 29639:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 625.577674] LNet: 29639:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 625.584308] LNet: 29639:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 625.597958] LNet: Message forwarding will be enabled in 2s [ 628.704133] LNet: Message forwarding enabled [ 629.069929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 629.090334] LNet: Message forwarding will stop in 2s to 6s [ 632.804436] LNet: Message forwarding disabled [ 633.627782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 633.661646] LNet: 29738:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 633.673682] LNet: 29738:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 633.681774] LNet: 29738:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 633.691725] LNet: Message forwarding will be enabled in 2s [ 633.822369] LNet: 29751:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 634.435670] Key type .llcrypt unregistered [ 634.437520] Key type ._llcrypt unregistered [ 647.364066] Key type ._llcrypt registered [ 647.365761] Key type .llcrypt registered [ 649.079042] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 19:04:24 (1790895864) [ 649.640113] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 649.707270] alg: No test for adler32 (adler32-zlib) [ 650.682603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 650.871477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 658.759185] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 19:04:34 (1790895874) [ 659.300158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 659.342375] LNet: 31153:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 659.374856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 659.489143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 659.556357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 659.622195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 659.690582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 659.741316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 659.778181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 659.821728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 659.879688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 659.955549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 660.007912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 660.046868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 660.104983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 660.165720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 660.230869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 660.287944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 660.403727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 660.478519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 660.568981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 660.630813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 660.704598] LNetError: 31197:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 668.418488] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 19:04:43 (1790895883) [ 668.632831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 668.725368] LNet: 31764:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 668.789741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 668.894636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 677.712767] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 19:04:53 (1790895893) [ 679.478844] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg403-server [ 681.593373] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 19:04:57 (1790895897) [ 681.686586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 681.737715] LNet: 32691:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 681.784769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 681.888482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 681.950540] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 682.979222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.204.[4-8]@tcp [ 683.018707] LNet: 32833:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 683.079692] LNetError: 32701:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.7@tcp is being used as a gateway but routing feature is not turned on [ 683.113153] LNetError: 32701:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.7@tcp has gone from up to down [ 690.022639] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 19:05:05 (1790895905) [ 692.284697] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg403-server [ 694.154529] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 19:05:09 (1790895909) [ 694.249468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 694.294407] LNet: 33764:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 694.304971] LNetError: 33764:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 694.336299] LNet: Removed LNI 192.168.204.3@tcp [ 694.366904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 694.501126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 694.540278] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 695.510488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.204.[4-9/2]@tcp [ 695.545965] LNet: 33889:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 695.557578] LNet: 33889:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 695.598519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.204.[4-9/2]@tcp [ 695.611197] LNetError: 33775:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.6@tcp is being used as a gateway but routing feature is not turned on [ 695.641125] LNetError: 33775:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 695.666873] LNetError: 33775:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp103 through 192.168.204.6@tcp has gone from up to down [ 695.683972] LNetError: 33775:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 703.833485] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 19:05:19 (1790895919) [ 704.121975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 704.160409] LNet: 34481:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 704.172065] LNetError: 34481:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 704.203172] LNet: Removed LNI 192.168.204.3@tcp [ 704.307177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 704.500674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 704.593141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 704.627937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 704.753232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 704.852638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 704.897585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 704.968384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 705.048097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 705.080654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 705.164971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 705.278954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 705.385757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 705.502148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 705.624856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 705.658838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 705.724453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 705.803604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 705.885929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 705.982426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 706.114314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 706.161411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 706.236616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 706.373394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 706.441114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 706.533766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 706.640437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 706.705857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 706.803496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 706.904552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 706.945946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 707.032708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 707.108112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 707.162181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 707.316865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 715.155566] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 19:05:30 (1790895930) [ 717.355653] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg403-server [ 719.150397] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 19:05:34 (1790895934) [ 721.215903] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg403-server [ 722.788742] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 19:05:38 (1790895938) [ 722.970881] LNet: 35883:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 722.983793] LNet: 35883:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 723.691943] Key type .llcrypt unregistered [ 723.695796] Key type ._llcrypt unregistered [ 730.912502] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 731.154950] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 731.160271] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 731.168373] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 733.741417] Key type ._llcrypt registered [ 733.744606] Key type .llcrypt registered [ 733.939614] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 733.954740] alg: No test for adler32 (adler32-zlib) [ 734.927832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 735.039490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 735.083785] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 735.808629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 735.847242] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 738.249562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 738.269545] LNet: Removed LNI 10.1.2.3@tcp [ 738.410296] LNet: 36599:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 738.416668] LNetError: 36599:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 738.429812] LNet: Removed LNI 192.168.204.3@tcp [ 738.944593] Key type .llcrypt unregistered [ 738.947877] Key type ._llcrypt unregistered [ 751.276367] Key type ._llcrypt registered [ 751.285434] Key type .llcrypt registered [ 752.925260] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 19:06:08 (1790895968) [ 753.617465] Key type .llcrypt unregistered [ 753.619410] Key type ._llcrypt unregistered [ 759.563572] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 759.841201] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 759.856194] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 759.868921] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 762.376514] Key type ._llcrypt registered [ 762.379269] Key type .llcrypt registered [ 762.569981] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 762.604353] alg: No test for adler32 (adler32-zlib) [ 763.582660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 763.666080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 763.708507] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 764.323199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 764.336245] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 766.857709] LNet: 38092:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 766.865781] LNetError: 38092:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 766.872776] LNet: Removed LNI 10.1.2.3@tcp [ 766.882467] LNet: Removed LNI 192.168.204.3@tcp [ 767.352266] Key type .llcrypt unregistered [ 767.354929] Key type ._llcrypt unregistered [ 780.226294] Key type ._llcrypt registered [ 780.227992] Key type .llcrypt registered [ 781.741273] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 19:06:37 (1790895997) [ 782.324347] Key type .llcrypt unregistered [ 782.328559] Key type ._llcrypt unregistered [ 789.412907] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 789.584360] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 789.598754] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 789.605241] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 792.329075] Key type ._llcrypt registered [ 792.332886] Key type .llcrypt registered [ 792.572090] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 792.597678] alg: No test for adler32 (adler32-zlib) [ 793.642415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 793.770749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 793.831396] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 796.361656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 796.389385] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 798.870323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 798.921628] LNet: Removed LNI 10.1.2.3@tcp [ 798.944364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 799.001349] LNet: Removed LNI 10.1.2.31@tcp [ 799.038984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 799.097537] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 800.151516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 800.177860] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 801.373468] LNet: 40005:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 801.387407] LNetError: 40005:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 801.400670] LNet: Removed LNI 10.1.2.31@tcp [ 801.416258] LNet: Removed LNI 10.1.2.3@tcp [ 802.298658] Key type .llcrypt unregistered [ 802.301333] Key type ._llcrypt unregistered [ 815.093298] Key type ._llcrypt registered [ 815.095737] Key type .llcrypt registered [ 816.728496] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 19:07:12 (1790896032) [ 817.183322] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 817.203704] alg: No test for adler32 (adler32-zlib) [ 818.123288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 818.194790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 818.224751] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 818.826709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 818.854893] LNet: 40957:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 818.859238] LNetError: 40957:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 818.868899] LNet: Removed LNI 192.168.204.3@tcp [ 818.886371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 818.943917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.204.3@tcp [ 818.972956] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 819.666394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.204.3@tcp [ 819.708416] LNet: Removed LNI 192.168.204.3@tcp [ 819.787203] LNet: 41092:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 820.453621] Key type .llcrypt unregistered [ 820.458556] Key type ._llcrypt unregistered [ 832.388802] Key type ._llcrypt registered [ 832.391798] Key type .llcrypt registered [ 833.753773] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 19:07:29 (1790896049) [ 836.007857] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg403-server [ 837.642580] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 19:07:33 (1790896053) [ 838.245525] Key type .llcrypt unregistered [ 838.248373] Key type ._llcrypt unregistered [ 844.656895] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 844.980134] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 844.984555] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 844.994602] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 847.593283] Key type ._llcrypt registered [ 847.594522] Key type .llcrypt registered [ 847.799496] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 847.817736] alg: No test for adler32 (adler32-zlib) [ 848.742475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 848.802129] LNet: Added LNI 192.168.204.3@tcp1 [8/256/0/180] [ 849.584676] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 851.685948] LNet: 42927:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 851.703624] LNetError: 42927:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 851.723644] LNet: Removed LNI 192.168.204.3@tcp1 [ 851.746284] LNet: Removed LNI 10.1.2.3@tcp2 [ 852.366614] Key type .llcrypt unregistered [ 852.372146] Key type ._llcrypt unregistered [ 865.513052] Key type ._llcrypt registered [ 865.514836] Key type .llcrypt registered [ 867.115486] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 19:08:02 (1790896082) [ 867.517988] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 867.542623] alg: No test for adler32 (adler32-zlib) [ 868.520956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 868.587628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 868.614852] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 869.548888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 869.573724] LNet: 43896:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 869.585881] LNetError: 43896:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 869.598733] LNet: Removed LNI 192.168.204.3@tcp [ 869.622943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 869.712542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 869.781571] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 876.965968] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 19:08:12 (1790896092) [ 877.106857] LNet: 44480:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 877.121660] LNetError: 44480:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 877.152569] LNet: Removed LNI 192.168.204.3@tcp [ 877.719469] Key type .llcrypt unregistered [ 877.722187] Key type ._llcrypt unregistered [ 884.602718] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 884.901083] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 884.916336] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 884.941204] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 887.560849] Key type ._llcrypt registered [ 887.565671] Key type .llcrypt registered [ 887.810889] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 887.848925] alg: No test for adler32 (adler32-zlib) [ 888.774919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 888.843512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 888.873403] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 889.813597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 889.838915] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 892.278764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 892.302737] LNet: 45250:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 892.308835] LNetError: 45250:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 892.316626] LNet: Removed LNI 10.1.2.3@tcp [ 892.330415] LNet: Removed LNI 192.168.204.3@tcp [ 892.346709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 892.394502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 892.433593] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 892.437843] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 893.153877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 893.201357] LNet: 45277:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 893.221720] LNetError: 45277:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 893.227811] LNetError: 45277:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 893.236966] LNet: Removed LNI 10.1.2.3@tcp [ 893.251607] LNet: Removed LNI 192.168.204.3@tcp [ 893.271125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 893.325297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 893.347822] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 893.353191] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 893.950692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 894.020584] LNet: Added LNI 192.168.204.3@tcp2 [8/256/0/180] [ 895.135987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 895.162491] LNet: 45440:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 895.169970] LNetError: 45440:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 895.175672] LNetError: 45440:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 895.183421] LNet: Removed LNI 10.1.2.3@tcp [ 895.186558] LNet: Removed LNI 192.168.204.3@tcp [ 895.194338] LNet: Removed LNI 192.168.204.3@tcp2 [ 895.221601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 895.287290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 895.324624] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 895.334640] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 895.341529] LNet: Added LNI 192.168.204.3@tcp2 [8/256/0/180] [ 903.458089] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 19:08:39 (1790896119) [ 903.560255] LNet: 46032:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 903.575959] LNetError: 46032:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 903.590712] LNetError: 46032:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 903.603638] LNet: Removed LNI 10.1.2.3@tcp [ 903.607051] LNet: Removed LNI 192.168.204.3@tcp [ 903.623136] LNet: Removed LNI 192.168.204.3@tcp2 [ 904.353785] Key type .llcrypt unregistered [ 904.357190] Key type ._llcrypt unregistered [ 910.771385] Key type ._llcrypt registered [ 910.778896] Key type .llcrypt registered [ 911.105548] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 911.125347] alg: No test for adler32 (adler32-zlib) [ 912.117945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 912.203531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 912.231415] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 913.006107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 913.033263] LNet: 46432:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 913.041788] LNetError: 46432:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 913.056692] LNet: Removed LNI 192.168.204.3@tcp [ 913.085970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 913.150670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 913.170930] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 913.601955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 913.635969] LNet: 46462:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 913.642890] LNetError: 46462:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 913.658943] LNet: Removed LNI 192.168.204.3@tcp [ 913.681226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 913.761106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 913.837588] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 914.341550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 914.383211] LNet: Removed LNI 192.168.204.3@tcp [ 914.409210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 914.484340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 914.519190] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 914.892187] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 914.988956] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 914.992176] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 914.996291] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 917.470432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 917.492388] LNet: 46566:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 917.500223] LNet: 46566:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 917.506662] LNetError: 46566:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 917.516046] LNetError: 46566:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 917.539958] LNet: Removed LNI 192.168.204.3@tcp [ 917.555376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 917.617416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 917.655096] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 924.703926] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 19:09:00 (1790896140) [ 925.021986] LNet: 47163:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 925.041508] LNetError: 47163:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 925.069607] LNet: Removed LNI 192.168.204.3@tcp [ 925.747811] Key type .llcrypt unregistered [ 925.753184] Key type ._llcrypt unregistered [ 932.306157] Key type ._llcrypt registered [ 932.309696] Key type .llcrypt registered [ 932.541895] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 932.552722] alg: No test for adler32 (adler32-zlib) [ 933.494881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 933.563752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 933.584989] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 934.279429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 934.308949] LNet: 47560:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 934.320885] LNetError: 47560:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 934.338126] LNet: Removed LNI 192.168.204.3@tcp [ 934.380552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 934.491386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 934.530773] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 941.328952] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 19:09:17 (1790896157) [ 941.713632] LNet: 48153:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 941.725456] LNetError: 48153:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 941.750578] LNet: Removed LNI 192.168.204.3@tcp [ 942.629812] Key type .llcrypt unregistered [ 942.631542] Key type ._llcrypt unregistered [ 950.298873] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 950.588057] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 950.604744] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 950.625674] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 953.270948] Key type ._llcrypt registered [ 953.273117] Key type .llcrypt registered [ 953.519762] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 953.559514] alg: No test for adler32 (adler32-zlib) [ 954.601432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 954.692497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 954.726466] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 955.987359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 956.019273] LNet: 48602:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 956.026056] LNetError: 48602:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 956.043339] LNet: Removed LNI 192.168.204.3@tcp [ 956.061891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 956.131593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 956.178915] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 963.571214] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 19:09:39 (1790896179) [ 963.828407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 963.855166] LNet: 49200:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 963.886733] LNetError: 49200:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 963.907548] LNet: Removed LNI 192.168.204.3@tcp [ 963.950075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 964.056977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 970.311565] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 19:09:46 (1790896186) [ 970.447684] LNet: 49776:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 971.047562] Key type .llcrypt unregistered [ 971.050321] Key type ._llcrypt unregistered [ 977.716219] Key type ._llcrypt registered [ 977.718916] Key type .llcrypt registered [ 978.043105] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 978.078712] alg: No test for adler32 (adler32-zlib) [ 979.037473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 979.114692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 979.147502] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 979.993093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 980.012498] LNet: 50178:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 980.020943] LNetError: 50178:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 980.035331] LNet: Removed LNI 192.168.204.3@tcp [ 980.054372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 980.132396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 980.165904] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 987.674756] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 19:10:03 (1790896203) [ 987.957052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 988.019396] LNet: 50772:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 988.038801] LNetError: 50772:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 988.071237] LNet: Removed LNI 192.168.204.3@tcp [ 988.093944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 988.223340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 994.800637] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 19:10:10 (1790896210) [ 996.875555] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 998.536607] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 19:10:14 (1790896214) [ 1000.377894] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 1002.813302] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 19:10:18 (1790896218) [ 1004.832470] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 1006.915750] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 19:10:22 (1790896222) [ 1009.160383] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1011.665241] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 19:10:26 (1790896226) [ 1013.680150] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1015.790865] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 19:10:31 (1790896231) [ 1015.948462] LNet: 52786:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1016.819769] Key type .llcrypt unregistered [ 1016.828879] Key type ._llcrypt unregistered [ 1025.727750] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1026.231754] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1026.238551] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1026.251416] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1028.909595] Key type ._llcrypt registered [ 1028.913505] Key type .llcrypt registered [ 1029.154190] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1029.171089] alg: No test for adler32 (adler32-zlib) [ 1030.124586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1030.247766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1030.286624] LNet: Added LNI 192.168.204.3@tcp [8/512/0/180] [ 1030.301841] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1030.307454] LNet: 53115:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1030.453964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1030.487461] LNet: 53134:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1030.504192] LNetError: 53134:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1030.511925] LNet: Removed LNI 10.1.2.3@tcp [ 1030.516577] LNetError: 53134:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1030.534716] LNet: Removed LNI 192.168.204.3@tcp [ 1030.568446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1030.666885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1030.696059] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 1030.766715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1030.788484] LNet: 53158:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1030.793797] LNetError: 53158:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1030.805292] LNet: Removed LNI 192.168.204.3@tcp [ 1030.822190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1030.906962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1030.931648] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 1030.941801] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1030.943918] LNet: 53166:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1031.064480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1031.087854] LNet: 53184:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1031.095899] LNetError: 53184:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1031.104328] LNet: Removed LNI 10.1.2.3@tcp [ 1031.107754] LNetError: 53184:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1031.133989] LNet: Removed LNI 192.168.204.3@tcp [ 1031.152648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1031.216991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1031.247476] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 1031.252647] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1031.255367] LNet: 53192:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1031.479279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1031.500839] LNet: 53218:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1031.515262] LNetError: 53218:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1031.523578] LNet: Removed LNI 10.1.2.3@tcp [ 1031.525863] LNetError: 53218:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1031.538276] LNet: Removed LNI 192.168.204.3@tcp [ 1031.558084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1031.639420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1031.678304] LNet: Added LNI 192.168.204.3@tcp [8/256/0/182] [ 1031.683110] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1031.789613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1031.816053] LNet: 53246:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1031.822334] LNetError: 53246:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1031.834479] LNet: Removed LNI 10.1.2.3@tcp [ 1031.843700] LNetError: 53246:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1031.867772] LNet: Removed LNI 192.168.204.3@tcp [ 1031.886915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1031.952464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1031.973318] LNet: Added LNI 192.168.204.3@tcp [8/256/0/182] [ 1031.977094] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1031.979573] LNet: 53254:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1032.048786] LNet: Removed LNI 10.1.2.3@tcp [ 1032.077098] LNet: Removed LNI 192.168.204.3@tcp [ 1032.702866] Key type .llcrypt unregistered [ 1032.705519] Key type ._llcrypt unregistered [ 1043.770558] Key type ._llcrypt registered [ 1043.772984] Key type .llcrypt registered [ 1045.220101] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 19:11:00 (1790896260) [ 1046.631880] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 1048.052464] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 19:11:04 (1790896264) [ 1048.484707] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1048.505876] alg: No test for adler32 (adler32-zlib) [ 1049.425788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1049.503249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1049.537027] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 1050.255072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.204.3 [ 1050.288506] LNet: 54520:0:(config.c:1740:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1050.302901] LNet: Removed LNI 192.168.204.3@tcp [ 1056.018660] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 19:11:12 (1790896272) [ 1056.265866] LNet: 55091:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1056.279419] LNetError: 55091:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1056.308411] LNet: Removed LNI 192.168.204.3@tcp [ 1056.789692] Key type .llcrypt unregistered [ 1056.793875] Key type ._llcrypt unregistered [ 1063.356259] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1063.606396] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1066.312138] Key type ._llcrypt registered [ 1066.314314] Key type .llcrypt registered [ 1066.506588] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1066.527618] alg: No test for adler32 (adler32-zlib) [ 1067.820518] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1074.442257] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 19:11:29 (1790896289) [ 1074.727902] LNet: 56003:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1074.745922] LNetError: 56003:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1074.779392] LNet: Removed LNI 10.1.2.3@tcp [ 1075.545734] Key type .llcrypt unregistered [ 1075.552556] Key type ._llcrypt unregistered [ 1082.662555] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1082.844797] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1082.847375] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1082.851189] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1085.631089] Key type ._llcrypt registered [ 1085.636713] Key type .llcrypt registered [ 1085.931583] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1085.953660] alg: No test for adler32 (adler32-zlib) [ 1086.988873] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1095.214321] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 19:11:50 (1790896310) [ 1097.052616] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1099.210665] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 19:11:54 (1790896314) [ 1099.481309] LNet: 57184:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1099.493086] LNetError: 57184:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1099.525841] LNet: Removed LNI 10.1.2.3@tcp [ 1100.177143] Key type .llcrypt unregistered [ 1100.178970] Key type ._llcrypt unregistered [ 1107.532465] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1107.799168] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1107.807224] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1107.816331] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1110.577410] Key type ._llcrypt registered [ 1110.580426] Key type .llcrypt registered [ 1110.795043] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1110.828824] alg: No test for adler32 (adler32-zlib) [ 1111.723242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1111.854131] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1119.997203] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 19:12:16 (1790896336) [ 1120.383372] LNet: 58415:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1120.387647] LNetError: 58415:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1120.401460] LNet: Removed LNI 10.1.2.3@tcp [ 1120.984594] Key type .llcrypt unregistered [ 1120.989263] Key type ._llcrypt unregistered [ 1127.222024] Key type ._llcrypt registered [ 1127.227679] Key type .llcrypt registered [ 1127.612811] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1127.632755] alg: No test for adler32 (adler32-zlib) [ 1128.663889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1128.704968] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 1138.050483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.103@tcp [ 1138.941824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.3@tcp -d 192.168.204.103@tcp -m GET -e local_interrupt -r 1 [ 1138.994337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.3@tcp -d 192.168.204.3@tcp -m GET -e local_interrupt -r 1 [ 1139.033160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.103@tcp [ 1139.061474] LNet: There was an unexpected network error while writing to 192.168.204.103: rc = -22 [ 1139.355239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1139.412551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1139.460838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1139.512189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1139.552742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1139.583769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1139.611801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1139.637512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1139.673839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1139.704574] LNet: 58853:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1139.740094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1139.772672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1139.824755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1139.859517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1139.886769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1139.919372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1139.956747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1140.512671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.3@tcp -d 192.168.204.103@tcp -m GET -e local_dropped -r 1 [ 1140.548385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.3@tcp -d 192.168.204.3@tcp -m GET -e local_dropped -r 1 [ 1140.597677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.103@tcp [ 1140.625970] LNet: There was an unexpected network error while writing to 192.168.204.103: rc = -22 [ 1140.877754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1140.928918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1141.000841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1141.024417] LNet: 1 local NIs in recovery (showing 1): 192.168.204.3@tcp [ 1141.043221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1141.084605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1141.129982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1141.185441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1141.229817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1141.283482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1141.329291] LNet: 58965:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1141.363969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1141.416464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1141.457554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1141.479642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1141.509810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1141.563134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1141.608828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1142.180183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.3@tcp -d 192.168.204.103@tcp -m GET -e local_aborted -r 1 [ 1142.213818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.3@tcp -d 192.168.204.3@tcp -m GET -e local_aborted -r 1 [ 1142.242799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.103@tcp [ 1142.257736] LNet: There was an unexpected network error while writing to 192.168.204.103: rc = -22 [ 1142.262314] LNet: Skipped 1 previous similar message [ 1142.456210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1142.479641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1142.504825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1142.539948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1142.571030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1142.602657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1142.630524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1142.654167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1142.676328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1142.695342] LNet: 59076:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1142.715439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1142.738948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1142.778652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1142.830126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1142.885936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1142.921961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1142.978876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1143.574828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.3@tcp -d 192.168.204.103@tcp -m GET -e local_no_route -r 1 [ 1143.627631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.3@tcp -d 192.168.204.3@tcp -m GET -e local_no_route -r 1 [ 1143.708740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.103@tcp [ 1143.954877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1143.980439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1144.006875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1144.033991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1144.058978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1144.082504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1144.109200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1144.133320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1144.164573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1144.191438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1144.218698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1144.246804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1144.269975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1144.303047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1144.342258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1144.404701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1144.915976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.3@tcp -d 192.168.204.103@tcp -m GET -e local_timeout -r 1 [ 1144.995870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.3@tcp -d 192.168.204.3@tcp -m GET -e local_timeout -r 1 [ 1145.032310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.103@tcp [ 1145.052812] LNet: There was an unexpected network error while writing to 192.168.204.103: rc = -22 [ 1145.058635] LNet: Skipped 1 previous similar message [ 1145.311969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1145.358713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1145.410816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1145.458422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1145.503499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1145.546046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1145.616447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1145.674832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1145.711253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1145.736419] LNet: 59298:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1145.741244] LNet: 59298:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1145.766771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1145.811398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1145.865992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1145.941923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1145.990726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1146.037616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1146.089457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1146.681530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.3@tcp -d 192.168.204.103@tcp -m GET -e local_error -r 1 [ 1146.714774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.3@tcp -d 192.168.204.3@tcp -m GET -e local_error -r 1 [ 1146.775714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.103@tcp [ 1147.021721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1147.053594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1147.083644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1147.108665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1147.149321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1147.180280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1147.205729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1147.236172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1147.273563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1147.313909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1147.355580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1147.410767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1147.459402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1147.505316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1147.556864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1147.626389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1147.917527] LNet: 59451:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1147.923884] LNetError: 59451:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1147.943724] LNet: Removed LNI 192.168.204.3@tcp [ 1148.357501] Key type .llcrypt unregistered [ 1148.360583] Key type ._llcrypt unregistered [ 1166.393523] Key type ._llcrypt registered [ 1166.395996] Key type .llcrypt registered [ 1167.727468] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 19:13:03 (1790896383) [ 1168.861459] Key type .llcrypt unregistered [ 1168.866305] Key type ._llcrypt unregistered [ 1175.219918] Key type ._llcrypt registered [ 1175.224351] Key type .llcrypt registered [ 1175.457969] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1175.471083] alg: No test for adler32 (adler32-zlib) [ 1176.457891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1176.497170] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 1185.588801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.103@tcp [ 1197.309628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1197.356906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1197.385776] LNet: Added LNI 192.168.204.3@tcp1 [8/256/0/180] [ 1198.899896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.3@tcp -d 192.168.204.103@tcp -m GET -e local_interrupt -r 1 [ 1198.949932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.3@tcp -d 192.168.204.103@tcp1 -m GET -e local_interrupt -r 1 [ 1198.995375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.3@tcp -d 192.168.204.3@tcp -m GET -e local_interrupt -r 1 [ 1199.024743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.3@tcp -d 192.168.204.3@tcp1 -m GET -e local_interrupt -r 1 [ 1199.057518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.3@tcp1 -d 192.168.204.103@tcp -m GET -e local_interrupt -r 1 [ 1199.089287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.3@tcp1 -d 192.168.204.103@tcp1 -m GET -e local_interrupt -r 1 [ 1199.129167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.3@tcp1 -d 192.168.204.3@tcp -m GET -e local_interrupt -r 1 [ 1199.172456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.3@tcp1 -d 192.168.204.3@tcp1 -m GET -e local_interrupt -r 1 [ 1199.219497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.103@tcp [ 1199.235903] LNet: There was an unexpected network error while writing to 192.168.204.103: rc = -22 [ 1199.241732] LNet: 1 local NIs in recovery (showing 1): 192.168.204.3@tcp [ 1199.463325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1199.496326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1199.522932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1199.560796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1199.594269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1199.619725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1199.645950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1199.692074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1199.715584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1199.729295] LNet: 60918:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1199.743704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1199.770729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1199.795190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1199.821422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1199.847502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1199.872916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1199.902753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1200.366483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.3@tcp -d 192.168.204.103@tcp -m GET -e local_dropped -r 1 [ 1200.393897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.3@tcp -d 192.168.204.103@tcp1 -m GET -e local_dropped -r 1 [ 1200.426461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.3@tcp -d 192.168.204.3@tcp -m GET -e local_dropped -r 1 [ 1200.464275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.3@tcp -d 192.168.204.3@tcp1 -m GET -e local_dropped -r 1 [ 1200.498142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.3@tcp1 -d 192.168.204.103@tcp -m GET -e local_dropped -r 1 [ 1200.554614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.3@tcp1 -d 192.168.204.103@tcp1 -m GET -e local_dropped -r 1 [ 1200.630064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.3@tcp1 -d 192.168.204.3@tcp -m GET -e local_dropped -r 1 [ 1200.670570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.3@tcp1 -d 192.168.204.3@tcp1 -m GET -e local_dropped -r 1 [ 1200.708278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.103@tcp [ 1200.727609] LNet: There was an unexpected network error while writing to 192.168.204.103: rc = -22 [ 1200.732396] LNet: Skipped 2 previous similar messages [ 1200.925577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1200.955222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1200.981469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1201.006526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1201.030650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1201.057207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1201.078153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1201.113840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1201.142916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1201.159771] LNet: 61041:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1201.175720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1201.199904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1201.227216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1201.256737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1201.284315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1201.314910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1201.363483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1201.879350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.3@tcp -d 192.168.204.103@tcp -m GET -e local_aborted -r 1 [ 1201.923689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.3@tcp -d 192.168.204.103@tcp1 -m GET -e local_aborted -r 1 [ 1201.985211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.3@tcp -d 192.168.204.3@tcp -m GET -e local_aborted -r 1 [ 1202.033964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.3@tcp -d 192.168.204.3@tcp1 -m GET -e local_aborted -r 1 [ 1202.065473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.3@tcp1 -d 192.168.204.103@tcp -m GET -e local_aborted -r 1 [ 1202.119332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.3@tcp1 -d 192.168.204.103@tcp1 -m GET -e local_aborted -r 1 [ 1202.149176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.3@tcp1 -d 192.168.204.3@tcp -m GET -e local_aborted -r 1 [ 1202.216551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.3@tcp1 -d 192.168.204.3@tcp1 -m GET -e local_aborted -r 1 [ 1202.272372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.103@tcp [ 1202.301836] LNet: There was an unexpected network error while writing to 192.168.204.103: rc = -22 [ 1202.311263] LNet: Skipped 2 previous similar messages [ 1202.552406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1202.584696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1202.616731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1202.646370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1202.672708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1202.698235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1202.722657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1202.749398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1202.780364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1202.795331] LNet: 61165:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1202.816589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1202.843888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1202.873803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1202.923695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1202.964542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1203.009293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1203.074126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1203.696633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.3@tcp -d 192.168.204.103@tcp -m GET -e local_no_route -r 1 [ 1203.729389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.3@tcp -d 192.168.204.103@tcp1 -m GET -e local_no_route -r 1 [ 1203.760209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.3@tcp -d 192.168.204.3@tcp -m GET -e local_no_route -r 1 [ 1203.787367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.3@tcp -d 192.168.204.3@tcp1 -m GET -e local_no_route -r 1 [ 1203.818391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.3@tcp1 -d 192.168.204.103@tcp -m GET -e local_no_route -r 1 [ 1203.846286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.3@tcp1 -d 192.168.204.103@tcp1 -m GET -e local_no_route -r 1 [ 1203.872827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.3@tcp1 -d 192.168.204.3@tcp -m GET -e local_no_route -r 1 [ 1203.900168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.3@tcp1 -d 192.168.204.3@tcp1 -m GET -e local_no_route -r 1 [ 1203.927686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.103@tcp [ 1204.112289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1204.145073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1204.174421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1204.205628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1204.253491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1204.296697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1204.331467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1204.357797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1204.394591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1204.429797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1204.482821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1204.523363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1204.563843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1204.606652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1204.674359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1204.734577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1205.262861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.3@tcp -d 192.168.204.103@tcp -m GET -e local_timeout -r 1 [ 1205.325754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.3@tcp -d 192.168.204.103@tcp1 -m GET -e local_timeout -r 1 [ 1205.371173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.3@tcp -d 192.168.204.3@tcp -m GET -e local_timeout -r 1 [ 1205.422955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.3@tcp -d 192.168.204.3@tcp1 -m GET -e local_timeout -r 1 [ 1205.455677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.3@tcp1 -d 192.168.204.103@tcp -m GET -e local_timeout -r 1 [ 1205.495015] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.3@tcp1 -d 192.168.204.103@tcp1 -m GET -e local_timeout -r 1 [ 1205.550963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.3@tcp1 -d 192.168.204.3@tcp -m GET -e local_timeout -r 1 [ 1205.596488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.3@tcp1 -d 192.168.204.3@tcp1 -m GET -e local_timeout -r 1 [ 1205.648173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.103@tcp [ 1205.665287] LNet: There was an unexpected network error while writing to 192.168.204.103: rc = -22 [ 1205.670275] LNet: Skipped 5 previous similar messages [ 1205.899179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1205.937505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1205.966674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1206.012586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1206.043175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1206.075946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1206.106526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1206.137573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1206.160659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1206.180381] LNet: 61411:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1206.188506] LNet: 61411:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1206.219625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1206.252419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1206.282386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1206.313709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1206.337767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1206.369076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1206.398174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1206.956187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.3@tcp -d 192.168.204.103@tcp -m GET -e local_error -r 1 [ 1206.989103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.3@tcp -d 192.168.204.103@tcp1 -m GET -e local_error -r 1 [ 1207.023461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.3@tcp -d 192.168.204.3@tcp -m GET -e local_error -r 1 [ 1207.050296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.3@tcp -d 192.168.204.3@tcp1 -m GET -e local_error -r 1 [ 1207.087226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.3@tcp1 -d 192.168.204.103@tcp -m GET -e local_error -r 1 [ 1207.129657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.3@tcp1 -d 192.168.204.103@tcp1 -m GET -e local_error -r 1 [ 1207.173659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.3@tcp1 -d 192.168.204.3@tcp -m GET -e local_error -r 1 [ 1207.229630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.3@tcp1 -d 192.168.204.3@tcp1 -m GET -e local_error -r 1 [ 1207.267698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.103@tcp [ 1207.529709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1207.571216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1207.616767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1207.663305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1207.722402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1207.756035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1207.790615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1207.853485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1207.911523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1207.941149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1207.966379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1207.991487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1208.044431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1208.075242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1208.101757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1208.128351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1213.990296] LNet: 61611:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1213.998935] LNetError: 61611:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1214.014894] LNet: Removed LNI 192.168.204.3@tcp [ 1214.042325] LNet: Removed LNI 192.168.204.3@tcp1 [ 1214.468073] Key type .llcrypt unregistered [ 1214.472303] Key type ._llcrypt unregistered [ 1233.357631] Key type ._llcrypt registered [ 1233.362400] Key type .llcrypt registered [ 1235.369384] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 19:14:10 (1790896450) [ 1236.311478] Key type .llcrypt unregistered [ 1236.323337] Key type ._llcrypt unregistered [ 1244.253507] Key type ._llcrypt registered [ 1244.266379] Key type .llcrypt registered [ 1244.672364] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1244.696950] alg: No test for adler32 (adler32-zlib) [ 1245.689817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1245.752528] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 1254.795588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.103@tcp [ 1255.584925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.3@tcp -d 192.168.204.103@tcp -m GET -e remote_dropped -r 1 [ 1255.623736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.3@tcp -d 192.168.204.3@tcp -m GET -e remote_dropped -r 1 [ 1255.676368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.103@tcp [ 1255.705815] LNet: There was an unexpected network error while writing to 192.168.204.103: rc = -22 [ 1256.017826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1256.055751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1256.076472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1256.105351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1256.149443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1256.187586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1256.224174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1256.253463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1256.272716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1256.291275] LNet: 62861:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1256.306392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1256.335571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1256.364418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1256.405212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1256.445960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1256.486225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1256.544187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1257.175529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.3@tcp -d 192.168.204.103@tcp -m GET -e remote_error -r 1 [ 1257.211678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.3@tcp -d 192.168.204.3@tcp -m GET -e remote_error -r 1 [ 1257.252356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.103@tcp [ 1257.271972] LNet: There was an unexpected network error while writing to 192.168.204.103: rc = -22 [ 1257.524412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1257.558292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1257.598704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1257.635612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1257.681688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1257.728112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1257.774582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1257.828828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1257.878670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1257.923516] LNet: 62972:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1257.965554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1258.012395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1258.060208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1258.114353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1258.186278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1258.247756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1258.339061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1259.012980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.3@tcp -d 192.168.204.103@tcp -m GET -e remote_timeout -r 1 [ 1259.094659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.3@tcp -d 192.168.204.3@tcp -m GET -e remote_timeout -r 1 [ 1259.134463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.103@tcp [ 1259.168268] LNet: There was an unexpected network error while writing to 192.168.204.103: rc = -22 [ 1259.450448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1259.486359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1259.510395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1259.546419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1259.579101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1259.605381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1259.645687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1259.688364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1259.744781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1259.780325] LNet: 63083:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1259.801555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1259.828891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1259.878340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1259.947623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1259.994474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1260.039655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1260.106392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1260.438742] LNet: 63124:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1260.454896] LNetError: 63124:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1260.467130] LNet: Removed LNI 192.168.204.3@tcp [ 1260.845472] Key type .llcrypt unregistered [ 1260.850906] Key type ._llcrypt unregistered [ 1280.617503] Key type ._llcrypt registered [ 1280.620634] Key type .llcrypt registered [ 1282.239211] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 19:14:57 (1790896497) [ 1283.241551] Key type .llcrypt unregistered [ 1283.244702] Key type ._llcrypt unregistered [ 1290.686405] Key type ._llcrypt registered [ 1290.689727] Key type .llcrypt registered [ 1290.949944] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1290.977835] alg: No test for adler32 (adler32-zlib) [ 1291.972104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1292.011679] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 1301.420909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.103@tcp [ 1312.481987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1312.520235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1312.570469] LNet: Added LNI 192.168.204.3@tcp1 [8/256/0/180] [ 1314.526799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.3@tcp -d 192.168.204.103@tcp -m GET -e remote_dropped -r 1 [ 1314.562800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.3@tcp -d 192.168.204.103@tcp1 -m GET -e remote_dropped -r 1 [ 1314.612577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.3@tcp -d 192.168.204.3@tcp -m GET -e remote_dropped -r 1 [ 1314.669924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.3@tcp -d 192.168.204.3@tcp1 -m GET -e remote_dropped -r 1 [ 1314.712233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.3@tcp1 -d 192.168.204.103@tcp -m GET -e remote_dropped -r 1 [ 1314.759671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.3@tcp1 -d 192.168.204.103@tcp1 -m GET -e remote_dropped -r 1 [ 1314.839789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.3@tcp1 -d 192.168.204.3@tcp -m GET -e remote_dropped -r 1 [ 1314.876759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.3@tcp1 -d 192.168.204.3@tcp1 -m GET -e remote_dropped -r 1 [ 1314.916742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.103@tcp [ 1314.937060] LNet: There was an unexpected network error while writing to 192.168.204.103: rc = -22 [ 1314.951916] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.103@tcp [ 1315.187511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1315.241920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1315.292517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1315.336942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1315.381658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1315.445427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1315.500648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1315.580406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1315.650842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1315.698957] LNet: 64601:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1315.719465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1315.752891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1315.796876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1315.839346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1315.904789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1315.954855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1316.019468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1316.078231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1316.851867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.3@tcp -d 192.168.204.103@tcp -m GET -e remote_error -r 1 [ 1316.882657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.3@tcp -d 192.168.204.103@tcp1 -m GET -e remote_error -r 1 [ 1316.941061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.3@tcp -d 192.168.204.3@tcp -m GET -e remote_error -r 1 [ 1316.988173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.3@tcp -d 192.168.204.3@tcp1 -m GET -e remote_error -r 1 [ 1317.072376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.3@tcp1 -d 192.168.204.103@tcp -m GET -e remote_error -r 1 [ 1317.155345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.3@tcp1 -d 192.168.204.103@tcp1 -m GET -e remote_error -r 1 [ 1317.252194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.3@tcp1 -d 192.168.204.3@tcp -m GET -e remote_error -r 1 [ 1317.339157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.3@tcp1 -d 192.168.204.3@tcp1 -m GET -e remote_error -r 1 [ 1317.434613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.103@tcp [ 1317.485506] LNet: There was an unexpected network error while writing to 192.168.204.103: rc = -22 [ 1317.493147] LNet: Skipped 2 previous similar messages [ 1317.855993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1317.915938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1317.967220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1318.031289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1318.070476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1318.105592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1318.135820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1318.177926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1318.220847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1318.262274] LNet: 64727:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1318.294500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1318.356484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1318.408654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1318.460083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1318.505605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1318.532565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1318.564426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1318.592440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1319.211148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.3@tcp -d 192.168.204.103@tcp -m GET -e remote_timeout -r 1 [ 1319.274469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.3@tcp -d 192.168.204.103@tcp1 -m GET -e remote_timeout -r 1 [ 1319.326208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.3@tcp -d 192.168.204.3@tcp -m GET -e remote_timeout -r 1 [ 1319.410660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.3@tcp -d 192.168.204.3@tcp1 -m GET -e remote_timeout -r 1 [ 1319.471630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.3@tcp1 -d 192.168.204.103@tcp -m GET -e remote_timeout -r 1 [ 1319.525495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.3@tcp1 -d 192.168.204.103@tcp1 -m GET -e remote_timeout -r 1 [ 1319.579831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.3@tcp1 -d 192.168.204.3@tcp -m GET -e remote_timeout -r 1 [ 1319.636564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.3@tcp1 -d 192.168.204.3@tcp1 -m GET -e remote_timeout -r 1 [ 1319.692265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.103@tcp [ 1319.726787] LNet: There was an unexpected network error while writing to 192.168.204.103: rc = -22 [ 1319.736584] LNet: Skipped 1 previous similar message [ 1319.945527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1320.015694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1320.057968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1320.088757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1320.130632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1320.166472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1320.194817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1320.227971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1320.281965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1320.305186] LNet: 64852:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1320.340780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1320.384709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1320.434764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1320.479204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1320.556718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1320.603508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1320.649239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1320.687302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1325.198291] LNet: 64930:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1325.208753] LNetError: 64930:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1325.215629] LNet: Removed LNI 192.168.204.3@tcp [ 1325.227874] LNet: Removed LNI 192.168.204.3@tcp1 [ 1325.644469] Key type .llcrypt unregistered [ 1325.654592] Key type ._llcrypt unregistered [ 1342.639454] Key type ._llcrypt registered [ 1342.648753] Key type .llcrypt registered [ 1344.179863] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 19:15:59 (1790896559) [ 1346.040380] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1347.826680] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 19:16:03 (1790896563) [ 1348.717323] Key type .llcrypt unregistered [ 1348.723760] Key type ._llcrypt unregistered [ 1354.804487] Key type ._llcrypt registered [ 1354.809408] Key type .llcrypt registered [ 1355.064541] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1355.081347] alg: No test for adler32 (adler32-zlib) [ 1356.086580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1356.112436] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 1364.875622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.103@tcp [ 1365.683965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.3@tcp -d 192.168.204.103@tcp -m GET -e network_timeout -r 1 [ 1365.776581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.3@tcp -d 192.168.204.3@tcp -m GET -e network_timeout -r 1 [ 1365.837306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.103@tcp [ 1365.870452] LNet: There was an unexpected network error while writing to 192.168.204.103: rc = -22 [ 1366.137744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1366.173040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1366.216218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1366.267312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1366.309427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1366.340909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1366.380328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1366.381431] LNet: 1 local NIs in recovery (showing 1): 192.168.204.3@tcp [ 1366.394947] LNet: There was an unexpected network error while writing to 192.168.204.3: rc = -22 [ 1366.426919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1366.499047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1366.537814] LNet: 66475:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1366.591640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1366.640463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1366.677595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1366.706582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1366.761927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1366.815856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1367.193679] LNet: 66514:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1367.208243] LNetError: 66514:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1367.230787] LNet: Removed LNI 192.168.204.3@tcp [ 1367.672490] Key type .llcrypt unregistered [ 1367.676068] Key type ._llcrypt unregistered [ 1388.783482] Key type ._llcrypt registered [ 1388.786838] Key type .llcrypt registered [ 1388.962779] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1388.994566] alg: No test for adler32 (adler32-zlib) [ 1389.896649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1389.918818] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 1397.108397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.103@tcp [ 1407.448729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1407.502925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1407.533754] LNet: Added LNI 192.168.204.3@tcp1 [8/256/0/180] [ 1408.689674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.3@tcp -d 192.168.204.103@tcp -m GET -e network_timeout -r 1 [ 1408.751765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.3@tcp -d 192.168.204.103@tcp1 -m GET -e network_timeout -r 1 [ 1408.822859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.3@tcp -d 192.168.204.3@tcp -m GET -e network_timeout -r 1 [ 1408.876569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.3@tcp -d 192.168.204.3@tcp1 -m GET -e network_timeout -r 1 [ 1408.936404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.3@tcp1 -d 192.168.204.103@tcp -m GET -e network_timeout -r 1 [ 1408.967606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.3@tcp1 -d 192.168.204.103@tcp1 -m GET -e network_timeout -r 1 [ 1409.027924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.3@tcp1 -d 192.168.204.3@tcp -m GET -e network_timeout -r 1 [ 1409.073478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.3@tcp1 -d 192.168.204.3@tcp1 -m GET -e network_timeout -r 1 [ 1409.130607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.103@tcp [ 1409.162908] LNet: There was an unexpected network error while writing to 192.168.204.103: rc = -22 [ 1409.335610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1409.362748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1409.376829] LNet: 1 local NIs in recovery (showing 1): 192.168.204.3@tcp [ 1409.379559] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.103@tcp [ 1409.398565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1409.423459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1409.451428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1409.477739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1409.501286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1409.570747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1409.631129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1409.658217] LNet: 67394:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1409.684958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1409.744548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1409.807815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1409.870357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1409.916451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1410.004498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1416.735333] LNet: There was an unexpected network error while writing to 192.168.204.103: rc = -22 [ 1417.165430] LNet: 67468:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1417.173426] LNetError: 67468:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1417.185656] LNet: Removed LNI 192.168.204.3@tcp [ 1417.196692] LNet: Removed LNI 192.168.204.3@tcp1 [ 1417.610284] Key type .llcrypt unregistered [ 1417.614139] Key type ._llcrypt unregistered [ 1436.996806] Key type ._llcrypt registered [ 1437.001601] Key type .llcrypt registered [ 1438.104356] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 19:17:34 (1790896654) [ 1438.431637] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1438.448702] alg: No test for adler32 (adler32-zlib) [ 1439.347254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1439.405802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1439.441133] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 1440.301629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1440.330082] LNet: Added LNI 192.168.204.3@tcp1 [8/256/0/180] [ 1441.190287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.3@tcp [ 1441.261720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1441.332032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1441.385997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1441.423132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1441.478522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1441.521411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.3@tcp [ 1441.540023] LNetError: 68568:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.204.3@tcp to recovery [ 1442.465832] LNet: 1 local NIs in recovery (showing 1): 192.168.204.3@tcp [ 1442.466085] LNet: There was an unexpected network error while writing to 192.168.204.3: rc = -22 [ 1444.576436] LNet: There was an unexpected network error while writing to 192.168.204.3: rc = -22 [ 1445.043657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1448.672558] LNet: There was an unexpected network error while writing to 192.168.204.3: rc = -22 [ 1449.663132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1449.715993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1449.817082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1449.842498] LNet: 68685:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1449.851642] LNetError: 68685:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1449.861578] LNet: Removed LNI 192.168.204.3@tcp [ 1449.874434] LNet: Removed LNI 192.168.204.3@tcp1 [ 1449.898815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1449.937725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1449.957036] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 1450.419351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1450.432859] LNet: Added LNI 192.168.204.3@tcp1 [8/256/0/180] [ 1451.126198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.3@tcp [ 1451.160869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1451.222654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1451.255445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1451.286726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1451.320498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1451.346788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1451.404912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.3@tcp [ 1451.433478] LNetError: 68957:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.204.3@tcp to recovery [ 1452.000892] LNet: There was an unexpected network error while writing to 192.168.204.3: rc = -22 [ 1453.959709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1457.120483] LNet: There was an unexpected network error while writing to 192.168.204.3: rc = -22 [ 1457.134085] LNet: Skipped 1 previous similar message [ 1461.985457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1462.010136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1462.028297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1469.389366] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 19:18:05 (1790896685) [ 1469.487518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1469.516265] LNet: 69663:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1469.522664] LNetError: 69663:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1469.527311] LNetError: 69663:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1469.543315] LNet: Removed LNI 192.168.204.3@tcp [ 1469.588162] LNet: Removed LNI 192.168.204.3@tcp1 [ 1469.670780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1469.740825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1469.793138] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 1470.575544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1470.594320] LNet: Added LNI 192.168.204.3@tcp1 [8/256/0/180] [ 1471.398678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.3@tcp [ 1471.541979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1471.587377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1471.644157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1471.685199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1471.715653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1471.757031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.3@tcp --health 0 [ 1471.879700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1472.800463] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.3@tcp [ 1472.802132] LNet: There was an unexpected network error while writing to 192.168.204.3: rc = -22 [ 1472.820053] LNet: Skipped 1 previous similar message [ 1486.254366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1486.457597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1486.510203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1486.549215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.3@tcp --health 500 [ 1486.697138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1492.290412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1492.342658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1492.373357] LNet: 70141:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1492.379564] LNetError: 70141:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1492.389754] LNetError: 70141:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1492.399925] LNet: Removed LNI 192.168.204.3@tcp [ 1492.410849] LNet: Removed LNI 192.168.204.3@tcp1 [ 1492.453857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1492.530843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1492.563047] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 1493.513719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1493.547028] LNet: Added LNI 192.168.204.3@tcp1 [8/256/0/180] [ 1494.534504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.3@tcp [ 1494.584828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1494.676531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1494.726943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1494.776632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1494.826128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1494.890904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1494.945954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.3@tcp --health 0 [ 1495.584728] LNet: There was an unexpected network error while writing to 192.168.204.3: rc = -22 [ 1495.598363] LNet: Skipped 3 previous similar messages [ 1496.402753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1506.828308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1506.856408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1506.893379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1513.132454] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 19:18:49 (1790896729) [ 1513.596589] LNet: 71188:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1513.607171] LNetError: 71188:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1513.616451] LNetError: 71188:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1513.623153] LNet: Removed LNI 192.168.204.3@tcp [ 1513.645221] LNet: Removed LNI 192.168.204.3@tcp1 [ 1514.258725] Key type .llcrypt unregistered [ 1514.263261] Key type ._llcrypt unregistered [ 1519.992615] Key type ._llcrypt registered [ 1519.995197] Key type .llcrypt registered [ 1520.221180] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1520.238520] alg: No test for adler32 (adler32-zlib) [ 1521.126525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1521.166342] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 1530.662754] Lustre: DEBUG MARKER: Initial discovery [ 1530.883935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.103@tcp [ 1533.436085] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1533.549519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1533.585633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.103@tcp [ 1533.622816] LNet: There was an unexpected network error while writing to 192.168.204.103: rc = -22 [ 1533.653827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1533.678531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.103@tcp -r 1 -m GET -l 3 [ 1533.708807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.204.103@tcp -r 1 -m GET -e local_error [ 1533.743747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.103@tcp -r 1 -m PUT -l 6 [ 1534.509685] LNet: 1 local NIs in recovery (showing 1): 192.168.204.3@tcp [ 1535.625896] Lustre: DEBUG MARKER: Force oleg403-client.virtnet to discover 192.168.204.103@tcp (in background) [ 1535.779877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.103@tcp [ 1537.980420] Lustre: DEBUG MARKER: Force oleg403-server to discover 192.168.204.3@tcp [ 1538.920743] LNet: There was an unexpected network error while writing to 192.168.204.103: rc = -22 [ 1541.344330] Lustre: DEBUG MARKER: Wait for 71942 [ 1542.432228] LNet: There was an unexpected network error while writing to 192.168.204.103: rc = -22 [ 1542.839675] Lustre: DEBUG MARKER: Finished wait on 71942 [ 1543.091275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1543.160294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1543.401025] LNet: 72399:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1543.407915] LNetError: 72399:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1543.418958] LNet: Removed LNI 192.168.204.3@tcp [ 1543.853677] Key type .llcrypt unregistered [ 1543.855549] Key type ._llcrypt unregistered [ 1560.914460] Key type ._llcrypt registered [ 1560.917193] Key type .llcrypt registered [ 1562.187863] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 19:19:37 (1790896777) [ 1562.731278] Key type .llcrypt unregistered [ 1562.736431] Key type ._llcrypt unregistered [ 1567.882607] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1568.046556] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1568.052364] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1568.064017] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1570.512872] Key type ._llcrypt registered [ 1570.515910] Key type .llcrypt registered [ 1570.654405] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1570.662530] alg: No test for adler32 (adler32-zlib) [ 1571.550465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1571.624742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1571.654681] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 1572.284493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1572.304338] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1579.266902] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 19:19:55 (1790896795) [ 1579.439697] LNet: 74470:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1579.458935] LNetError: 74470:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1579.467834] LNet: Removed LNI 10.1.2.3@tcp [ 1579.496339] LNet: Removed LNI 192.168.204.3@tcp [ 1580.119857] Key type .llcrypt unregistered [ 1580.123269] Key type ._llcrypt unregistered [ 1586.145844] Key type ._llcrypt registered [ 1586.147961] Key type .llcrypt registered [ 1586.333465] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1586.359831] alg: No test for adler32 (adler32-zlib) [ 1587.298222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1587.368502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1587.401549] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 1588.136616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1588.151919] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1589.230557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.204.3@tcp [ 1590.306161] LNetError: Unexpected error -22 connecting to 192.168.204.3@tcp at host 192.168.204.3:988 [ 1595.474349] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 19:20:11 (1790896811) [ 1595.581301] LNet: 75595:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1595.594274] LNetError: 75595:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1595.601485] LNet: Removed LNI 10.1.2.3@tcp [ 1595.624170] LNet: Removed LNI 192.168.204.3@tcp [ 1596.427573] Key type .llcrypt unregistered [ 1596.429729] Key type ._llcrypt unregistered [ 1603.492785] Key type ._llcrypt registered [ 1603.494742] Key type .llcrypt registered [ 1603.664049] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1603.678595] alg: No test for adler32 (adler32-zlib) [ 1604.613497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1604.683262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1604.706501] LNet: Added LNI 192.168.204.3@tcp1 [8/256/0/180] [ 1605.401589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1605.427621] LNet: Added LNI 192.168.204.3@tcp2 [8/256/0/180] [ 1606.334719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.204.3@tcp1 --nid 192.168.204.3@tcp2 [ 1610.499530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.3@tcp1 192.168.204.3@tcp2 [ 1610.521694] LNetError: 76438:0:(lib-move.c:2094:lnet_handle_find_routed_path()) Unable to send message from 192.168.204.3@tcp1 to 192.168.204.3@tcp2 - Route table may be misconfigured [ 1610.574832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.3@tcp2 192.168.204.3@tcp1 [ 1618.844603] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 19:20:34 (1790896834) [ 1619.030989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1619.075296] LNet: 77011:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1619.088932] LNetError: 77011:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1619.099354] LNet: Removed LNI 192.168.204.3@tcp1 [ 1620.136858] LNet: Removed LNI 192.168.204.3@tcp2 [ 1620.177683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1620.257637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1620.296721] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 1621.341722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1621.362315] LNet: Added LNI 192.168.204.3@tcp1 [8/256/0/180] [ 1622.353431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.3@tcp [ 1622.391690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.3@tcp 192.168.204.3@tcp [ 1622.428924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.3@tcp1 192.168.204.3@tcp1 [ 1622.493976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.3@tcp -d 192.168.204.3@tcp -e network_timeout [ 1622.558355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.3@tcp -d 192.168.204.3@tcp1 -e network_timeout [ 1622.594405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.3@tcp1 -d 192.168.204.3@tcp -e network_timeout [ 1622.644456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.3@tcp1 -d 192.168.204.3@tcp1 -e network_timeout [ 1622.689909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.3@tcp [ 1622.721542] LNet: There was an unexpected network error while writing to 192.168.204.3: rc = -22 [ 1622.744731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1622.858546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1622.936724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1623.328310] LNet: 1 local NIs in recovery (showing 1): 192.168.204.3@tcp [ 1628.887849] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 19:20:44 (1790896844) [ 1629.095634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1629.145633] LNet: 77874:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1629.153624] LNetError: 77874:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1629.159221] LNetError: 77874:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1629.166260] LNet: Removed LNI 192.168.204.3@tcp [ 1630.183410] LNet: Removed LNI 192.168.204.3@tcp1 [ 1630.218344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1630.327479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1630.718703] LNet: 77911:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1631.178946] Key type .llcrypt unregistered [ 1631.181964] Key type ._llcrypt unregistered [ 1642.224535] Key type ._llcrypt registered [ 1642.226234] Key type .llcrypt registered [ 1643.590270] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 19:20:59 (1790896859) [ 1643.914437] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1643.927781] alg: No test for adler32 (adler32-zlib) [ 1644.836850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1646.108887] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1647.802542] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 19:21:03 (1790896863) [ 1648.010840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1648.032901] LNet: 79008:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1648.056616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1648.117641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1648.154634] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 1648.942900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1648.963769] LNet: Added LNI 192.168.204.3@tcp1 [8/256/0/180] [ 1649.833495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.3@tcp [ 1649.877471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.3@tcp1 [ 1649.929864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.3@tcp1 [ 1655.761395] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 19:21:11 (1790896871) [ 1657.626142] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg403-server [ 1659.335861] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 19:21:15 (1790896875) [ 1661.129697] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg403-server [ 1663.149896] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 19:21:18 (1790896878) [ 1664.916948] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg403-server [ 1666.335242] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 19:21:22 (1790896882) [ 1667.955783] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg403-server [ 1669.705516] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 19:21:25 (1790896885) [ 1671.251749] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg403-server [ 1672.940638] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 19:21:28 (1790896888) [ 1675.066452] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg403-server [ 1676.829597] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 19:21:32 (1790896892) [ 1678.764361] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg403-server [ 1680.402171] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 19:21:36 (1790896896) [ 1682.094279] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg403-server [ 1683.846167] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 19:21:39 (1790896899) [ 1685.474527] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg403-server [ 1687.361204] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 19:21:43 (1790896903) [ 1687.462617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1687.489456] LNet: 82883:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1687.526431] LNetError: 82883:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1687.538467] LNet: Removed LNI 192.168.204.3@tcp [ 1688.619667] LNet: Removed LNI 192.168.204.3@tcp1 [ 1688.670713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1688.730634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1688.789668] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 1689.491369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1689.632532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.3@tcp [ 1689.884732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1689.918463] LNet: 83033:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1689.927650] LNetError: 83033:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1689.932610] LNetError: 83033:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1689.944514] LNet: Removed LNI 192.168.204.3@tcp [ 1689.975795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1690.047733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1690.072752] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 1691.163531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.3@tcp [ 1691.481825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1691.519345] LNetError: 83178:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1692.589637] LNet: Removed LNI 192.168.204.3@tcp [ 1692.625381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1692.751789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1692.801168] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 1693.744905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1693.847611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.3@tcp [ 1694.052482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1694.070881] LNet: 83326:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1694.076596] LNet: 83326:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1694.082922] LNetError: 83326:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1694.095659] LNet: Removed LNI 192.168.204.3@tcp [ 1694.122051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1694.184671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1694.226990] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 1695.010462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.3@tcp [ 1695.158501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1696.238524] LNet: Removed LNI 192.168.204.3@tcp [ 1696.260692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1696.344911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1696.379565] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 1697.196850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1697.290727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.3@tcp [ 1697.564287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1698.670485] LNet: Removed LNI 192.168.204.3@tcp [ 1698.712585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1698.852086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1698.895044] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 1699.837320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.3@tcp [ 1700.192573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1700.229174] LNet: 83764:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1700.240304] LNet: 83764:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1700.246469] LNetError: 83764:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1700.253549] LNetError: 83764:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1700.273839] LNet: Removed LNI 192.168.204.3@tcp [ 1700.312215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1700.418487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1700.456892] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 1701.527417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1701.639369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.3@tcp [ 1701.956399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1703.020421] LNet: Removed LNI 192.168.204.3@tcp [ 1703.070578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1703.164524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1703.199051] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 1704.282484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.3@tcp [ 1704.510727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1705.583831] LNet: Removed LNI 192.168.204.3@tcp [ 1705.634857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1705.689510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1705.729503] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 1706.425485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1706.505720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.3@tcp [ 1706.848536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1707.949804] LNet: Removed LNI 192.168.204.3@tcp [ 1708.003514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1708.109604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1708.141103] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 1708.984182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.3@tcp [ 1709.302603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1709.336714] LNet: 84350:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1709.348560] LNet: 84350:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1709.355867] LNetError: 84350:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1709.363120] LNetError: 84350:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1709.383347] LNet: Removed LNI 192.168.204.3@tcp [ 1709.412698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1709.494746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1709.522914] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 1710.411672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1710.547181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.3@tcp [ 1710.949361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1712.038956] LNet: Removed LNI 192.168.204.3@tcp [ 1712.069802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1712.193750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1712.229389] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 1712.898859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.3@tcp [ 1713.229548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1714.283787] LNet: Removed LNI 192.168.204.3@tcp [ 1714.340412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1714.443660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1714.486389] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 1715.195809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1715.255731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1715.299872] LNet: Removed LNI 192.168.204.3@tcp [ 1715.329748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1715.390660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1715.409087] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 1716.139940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1716.196716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1716.226492] LNet: Removed LNI 192.168.204.3@tcp [ 1716.246475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1716.294881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1716.392880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1716.449391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1716.520654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1716.633981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1716.683845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1716.753435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1716.821532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1716.857715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1716.914446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1716.935800] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 1717.881521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1717.920786] LNet: Removed LNI 192.168.204.3@tcp [ 1717.950535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1718.084380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1725.303990] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 19:22:20 (1790896940) [ 1725.449060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1725.482396] LNet: 85662:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1725.493624] LNet: 85662:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 1725.523133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1725.598961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1725.632548] LNet: Added LNI 192.168.204.3@tcp231 [8/256/0/180] [ 1726.938223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1726.986291] LNetError: 85805:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1727.000608] LNetError: 85805:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1727.031692] LNet: Removed LNI 192.168.204.3@tcp231 [ 1727.052876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1727.173992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1727.216219] LNet: Added LNI 192.168.204.3@tcp231 [8/256/0/180] [ 1727.224121] LNet: 85813:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1727.747329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1727.783954] LNet: Removed LNI 192.168.204.3@tcp231 [ 1727.810226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1727.852942] LNet: Added LNI 192.168.204.3@tcp231 [8/256/0/180] [ 1728.978232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1729.015321] LNet: Removed LNI 192.168.204.3@tcp231 [ 1729.046323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1729.125377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1729.168516] LNet: Added LNI 192.168.204.3@tcp231 [8/256/0/0] [ 1729.174085] LNet: 85966:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1729.755078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1729.794692] LNet: Removed LNI 192.168.204.3@tcp231 [ 1729.810081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1729.834071] LNet: Added LNI 192.168.204.3@tcp231 [8/256/0/0] [ 1730.972601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1731.024097] LNet: Removed LNI 192.168.204.3@tcp231 [ 1731.050106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1731.122750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1731.167894] LNet: Added LNI 192.168.204.3@tcp231 [8/256/0/60] [ 1731.186310] LNet: 86118:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1731.679403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1731.726299] LNet: Removed LNI 192.168.204.3@tcp231 [ 1731.754318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1731.785994] LNet: Added LNI 192.168.204.3@tcp231 [8/256/0/60] [ 1732.950312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1732.999560] LNet: Removed LNI 192.168.204.3@tcp231 [ 1733.042721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1733.142950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1733.170827] LNet: Added LNI 192.168.204.3@tcp231 [8/256/0/180] [ 1733.680707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1733.739228] LNet: Removed LNI 192.168.204.3@tcp231 [ 1733.775195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1733.813328] LNet: Added LNI 192.168.204.3@tcp231 [8/256/0/180] [ 1741.440230] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 19:22:37 (1790896957) [ 1741.516142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1741.555082] LNet: Removed LNI 192.168.204.3@tcp231 [ 1741.575656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1741.643870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1741.662937] LNet: Added LNI 192.168.204.3@tcp232 [8/256/0/180] [ 1742.321341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1742.366755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1742.395614] LNet: Removed LNI 192.168.204.3@tcp232 [ 1742.416128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1742.449186] LNet: Added LNI 192.168.204.3@tcp232 [8/256/0/180] [ 1743.255378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1743.285696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1743.319633] LNet: Removed LNI 192.168.204.3@tcp232 [ 1743.346329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1743.385683] LNet: Added LNI 192.168.204.3@tcp232 [8/256/0/180] [ 1744.015712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1744.061692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1744.087467] LNet: Removed LNI 192.168.204.3@tcp232 [ 1744.104591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1744.128739] LNet: Added LNI 192.168.204.3@tcp232 [8/256/0/180] [ 1744.807114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1744.849240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1744.882712] LNet: Removed LNI 192.168.204.3@tcp232 [ 1744.905600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1751.054481] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 19:22:46 (1790896966) [ 1752.148126] Key type .llcrypt unregistered [ 1752.152478] Key type ._llcrypt unregistered [ 1759.501438] Key type ._llcrypt registered [ 1759.504329] Key type .llcrypt registered [ 1759.660835] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1759.677522] alg: No test for adler32 (adler32-zlib) [ 1760.610256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1760.631565] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 1768.490539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.103@tcp [ 1779.220073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1779.276348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1779.322286] LNet: Added LNI 192.168.204.3@tcp1 [8/256/0/180] [ 1780.905836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.3@tcp -d 192.168.204.103@tcp -m GET -e local_interrupt -r 2 [ 1780.985196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.3@tcp -d 192.168.204.103@tcp1 -m GET -e local_interrupt -r 2 [ 1781.067897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.3@tcp -d 192.168.204.3@tcp -m GET -e local_interrupt -r 2 [ 1781.132513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.3@tcp -d 192.168.204.3@tcp1 -m GET -e local_interrupt -r 2 [ 1781.192640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.3@tcp1 -d 192.168.204.103@tcp -m GET -e local_interrupt -r 2 [ 1781.272395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.3@tcp1 -d 192.168.204.103@tcp1 -m GET -e local_interrupt -r 2 [ 1781.404703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.3@tcp1 -d 192.168.204.3@tcp -m GET -e local_interrupt -r 2 [ 1781.497834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.3@tcp1 -d 192.168.204.3@tcp1 -m GET -e local_interrupt -r 2 [ 1781.553875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.3@tcp 192.168.204.103@tcp [ 1781.593756] LNet: There was an unexpected network error while writing to 192.168.204.103: rc = -22 [ 1781.604797] LNet: 1 local NIs in recovery (showing 1): 192.168.204.3@tcp [ 1781.869219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1781.912234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1781.984736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1782.030437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1782.075535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1782.156172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1782.222812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1782.266618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1782.320873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1782.358125] LNet: 88771:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1782.404919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1782.433780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1782.490494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1782.568103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1782.645732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1782.707555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1783.471825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.3@tcp -d 192.168.204.103@tcp -m GET -e local_dropped -r 2 [ 1783.569437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.3@tcp -d 192.168.204.103@tcp1 -m GET -e local_dropped -r 2 [ 1783.636756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.3@tcp -d 192.168.204.3@tcp -m GET -e local_dropped -r 2 [ 1783.705716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.3@tcp -d 192.168.204.3@tcp1 -m GET -e local_dropped -r 2 [ 1783.807472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.3@tcp1 -d 192.168.204.103@tcp -m GET -e local_dropped -r 2 [ 1783.876840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.3@tcp1 -d 192.168.204.103@tcp1 -m GET -e local_dropped -r 2 [ 1783.967915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.3@tcp1 -d 192.168.204.3@tcp -m GET -e local_dropped -r 2 [ 1784.039346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.3@tcp1 -d 192.168.204.3@tcp1 -m GET -e local_dropped -r 2 [ 1784.110147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.3@tcp 192.168.204.103@tcp [ 1784.157738] LNet: There was an unexpected network error while writing to 192.168.204.103: rc = -22 [ 1784.576450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1784.690820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1784.740909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1784.784203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1784.854502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1784.933953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1784.984348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1785.085632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1785.138686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1785.213671] LNet: 88893:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1785.248782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1785.372987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1785.479722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1785.566312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1785.688276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1785.781347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1786.820663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.3@tcp -d 192.168.204.103@tcp -m GET -e local_aborted -r 2 [ 1786.898084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.3@tcp -d 192.168.204.103@tcp1 -m GET -e local_aborted -r 2 [ 1786.967565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.3@tcp -d 192.168.204.3@tcp -m GET -e local_aborted -r 2 [ 1787.051095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.3@tcp -d 192.168.204.3@tcp1 -m GET -e local_aborted -r 2 [ 1787.105607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.3@tcp1 -d 192.168.204.103@tcp -m GET -e local_aborted -r 2 [ 1787.133577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.3@tcp1 -d 192.168.204.103@tcp1 -m GET -e local_aborted -r 2 [ 1787.168730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.3@tcp1 -d 192.168.204.3@tcp -m GET -e local_aborted -r 2 [ 1787.207524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.3@tcp1 -d 192.168.204.3@tcp1 -m GET -e local_aborted -r 2 [ 1787.259398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.3@tcp 192.168.204.103@tcp [ 1787.292318] LNet: There was an unexpected network error while writing to 192.168.204.103: rc = -22 [ 1787.693906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1787.753059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1787.818278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1787.864686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1787.938768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1787.987690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1788.040878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1788.130854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1788.186610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1788.223831] LNet: 89016:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1788.254144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1788.311419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1788.376289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1788.433586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1788.501913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1788.555880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1789.433207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.3@tcp -d 192.168.204.103@tcp -m GET -e local_no_route -r 2 [ 1789.489919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.3@tcp -d 192.168.204.103@tcp1 -m GET -e local_no_route -r 2 [ 1789.559627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.3@tcp -d 192.168.204.3@tcp -m GET -e local_no_route -r 2 [ 1789.614882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.3@tcp -d 192.168.204.3@tcp1 -m GET -e local_no_route -r 2 [ 1789.686128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.3@tcp1 -d 192.168.204.103@tcp -m GET -e local_no_route -r 2 [ 1789.759627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.3@tcp1 -d 192.168.204.103@tcp1 -m GET -e local_no_route -r 2 [ 1789.830524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.3@tcp1 -d 192.168.204.3@tcp -m GET -e local_no_route -r 2 [ 1789.888149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.3@tcp1 -d 192.168.204.3@tcp1 -m GET -e local_no_route -r 2 [ 1789.960150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.3@tcp 192.168.204.103@tcp [ 1789.993731] LNet: There was an unexpected network error while writing to 192.168.204.103: rc = -22 [ 1790.409861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1790.442516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1790.472962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1790.521288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1790.581916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1790.646075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1790.674979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1790.716637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1790.761389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1790.800369] LNet: 89139:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1790.823519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1790.855879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1790.899395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1790.948438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1791.001465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1791.045429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1791.821269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.3@tcp -d 192.168.204.103@tcp -m GET -e local_timeout -r 2 [ 1791.859601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.3@tcp -d 192.168.204.103@tcp1 -m GET -e local_timeout -r 2 [ 1791.916804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.3@tcp -d 192.168.204.3@tcp -m GET -e local_timeout -r 2 [ 1791.981205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.3@tcp -d 192.168.204.3@tcp1 -m GET -e local_timeout -r 2 [ 1792.021392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.3@tcp1 -d 192.168.204.103@tcp -m GET -e local_timeout -r 2 [ 1792.069192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.3@tcp1 -d 192.168.204.103@tcp1 -m GET -e local_timeout -r 2 [ 1792.114897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.3@tcp1 -d 192.168.204.3@tcp -m GET -e local_timeout -r 2 [ 1792.173785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.3@tcp1 -d 192.168.204.3@tcp1 -m GET -e local_timeout -r 2 [ 1792.213218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.3@tcp 192.168.204.103@tcp [ 1792.414958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1792.445695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1792.469683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1792.499333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1792.523764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1792.554536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1792.597748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1792.628401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1792.669595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1792.707602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1792.743433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1792.779984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1792.825655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1792.877873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1792.941749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1793.519883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.3@tcp -d 192.168.204.103@tcp -m GET -e remote_dropped -r 2 [ 1793.554543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.3@tcp -d 192.168.204.103@tcp1 -m GET -e remote_dropped -r 2 [ 1793.589607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.3@tcp -d 192.168.204.3@tcp -m GET -e remote_dropped -r 2 [ 1793.636592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.3@tcp -d 192.168.204.3@tcp1 -m GET -e remote_dropped -r 2 [ 1793.699828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.3@tcp1 -d 192.168.204.103@tcp -m GET -e remote_dropped -r 2 [ 1793.743734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.3@tcp1 -d 192.168.204.103@tcp1 -m GET -e remote_dropped -r 2 [ 1793.781079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.3@tcp1 -d 192.168.204.3@tcp -m GET -e remote_dropped -r 2 [ 1793.841323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.3@tcp1 -d 192.168.204.3@tcp1 -m GET -e remote_dropped -r 2 [ 1793.884882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.3@tcp 192.168.204.103@tcp [ 1793.912060] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.103@tcp [ 1794.205710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1794.260845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1794.302822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1794.372434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1794.429789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1794.468671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1794.518709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1794.569937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1794.624507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1794.685554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1794.767381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1794.823426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1794.876706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1794.950233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1794.992724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1799.410309] LNet: 89457:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1799.426502] LNetError: 89457:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1799.437091] LNet: Removed LNI 192.168.204.3@tcp [ 1799.450856] LNet: Removed LNI 192.168.204.3@tcp1 [ 1799.857594] Key type .llcrypt unregistered [ 1799.861779] Key type ._llcrypt unregistered [ 1817.781160] Key type ._llcrypt registered [ 1817.783386] Key type .llcrypt registered [ 1819.237537] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 19:23:54 (1790897034) [ 1819.730938] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1819.750900] alg: No test for adler32 (adler32-zlib) [ 1820.712467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1820.796477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1820.829288] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 1821.810358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1821.849575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.3@tcp [ 1821.876185] LNetError: 90435:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.204.3@tcp to recovery [ 1821.977455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.204.3@tcp [ 1822.080412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1828.479711] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 19:24:04 (1790897044) [ 1828.600305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1828.659024] LNet: 91018:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1828.666217] LNetError: 91018:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1828.696829] LNet: Removed LNI 192.168.204.3@tcp [ 1828.730971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1828.821532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1828.850797] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 1829.606068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1829.639881] LNet: Added LNI 192.168.204.3@tcp1 [8/256/0/180] [ 1830.562696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.204.3@tcp [ 1830.600523] LNetError: 91290:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.204.3@tcp to recovery [ 1830.816461] LNet: 1 local NIs in recovery (showing 1): 192.168.204.3@tcp [ 1830.821129] LNet: There was an unexpected network error while writing to 192.168.204.3: rc = -22 [ 1831.893378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.204.3@tcp [ 1839.095220] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 19:24:14 (1790897054) [ 1840.878518] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg403-server [ 1842.533682] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 19:24:18 (1790897058) [ 1844.667492] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg403-server [ 1846.625329] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 19:24:22 (1790897062) [ 1849.386957] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg403-server [ 1851.177219] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 19:24:26 (1790897066) [ 1851.297295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1851.344228] LNet: 92909:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1851.357286] LNetError: 92909:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1851.363339] LNet: Removed LNI 192.168.204.3@tcp [ 1851.390980] LNet: Removed LNI 192.168.204.3@tcp1 [ 1851.456974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1851.545855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1851.579442] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 1852.484978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1852.555893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1852.591935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1852.691295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1852.735647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1852.823347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1860.361243] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 19:24:35 (1790897075) [ 1860.471022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1860.531508] LNet: 93638:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1860.544087] LNetError: 93638:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1860.558875] LNetError: 93638:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1860.585393] LNet: Removed LNI 192.168.204.3@tcp [ 1860.635117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1860.761301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1860.813176] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 1868.578542] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 19:24:44 (1790897084) [ 1868.673935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1868.698166] LNet: 94343:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1868.704223] LNetError: 94343:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1868.720104] LNet: Removed LNI 192.168.204.3@tcp [ 1868.743059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1868.815100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1868.839149] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 1875.766803] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 19:24:51 (1790897091) [ 1877.319767] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1878.991266] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 19:24:54 (1790897094) [ 1879.640749] LNet: 95383:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1879.650752] LNetError: 95383:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1879.663451] LNet: Removed LNI 192.168.204.3@tcp [ 1880.250712] Key type .llcrypt unregistered [ 1880.253340] Key type ._llcrypt unregistered [ 1885.569951] Key type ._llcrypt registered [ 1885.574175] Key type .llcrypt registered [ 1885.818242] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1885.840915] alg: No test for adler32 (adler32-zlib) [ 1886.765622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1886.815797] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 1895.009587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.103@tcp [ 1900.145877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.204.103@tcp [ 1900.668222] LNet: 95799:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1900.687437] LNetError: 95799:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1900.718639] LNet: Removed LNI 192.168.204.3@tcp [ 1901.238767] Key type .llcrypt unregistered [ 1901.246610] Key type ._llcrypt unregistered [ 1913.230347] Key type ._llcrypt registered [ 1913.237495] Key type .llcrypt registered [ 1915.015835] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1916.588963] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1918.431378] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 19:25:34 (1790897134) [ 1920.150425] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1921.895322] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 19:25:37 (1790897137) [ 1923.813385] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg403-server [ 1926.115821] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 19:25:41 (1790897141) [ 1928.017122] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg403-server [ 1929.967317] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 19:25:45 (1790897145) [ 1930.928280] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1930.988088] alg: No test for adler32 (adler32-zlib) [ 1932.276826] Lustre: Lustre: Build Version: 2.17.58_63_g3705aed [ 1932.620360] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 1934.328232] Key type lgssc registered [ 1936.064902] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1959.842805] Key type lgssc unregistered [ 1960.110914] LNet: 98983:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1960.124513] LNetError: 98983:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1960.137053] LNet: Removed LNI 192.168.204.3@tcp [ 1961.095561] Key type .llcrypt unregistered [ 1961.097868] Key type ._llcrypt unregistered [ 1978.294322] Key type ._llcrypt registered [ 1978.295963] Key type .llcrypt registered [ 1979.966662] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 19:26:35 (1790897195) [ 1980.484550] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1980.502582] alg: No test for adler32 (adler32-zlib) [ 1981.449215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1981.589861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1981.664535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1981.730379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1981.781433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1988.905450] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 19:26:44 (1790897204) [ 1989.056759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1989.089756] LNet: 100403:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1989.156795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1989.217480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1989.241705] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 1990.174939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1990.200175] LNet: 100550:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1990.205517] LNetError: 100550:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1990.222687] LNet: Removed LNI 192.168.204.3@tcp [ 1990.239898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1990.286538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1990.311466] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 1990.974562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1990.996825] LNetError: 100578:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1991.011691] LNet: Removed LNI 192.168.204.3@tcp [ 1991.044472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1991.143704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1991.178076] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 1998.229613] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 19:26:53 (1790897213) [ 1998.317950] LNet: 101159:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1998.334930] LNet: 101159:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1998.346939] LNetError: 101159:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1998.366153] LNet: Removed LNI 192.168.204.3@tcp [ 1999.026741] Key type .llcrypt unregistered [ 1999.030120] Key type ._llcrypt unregistered [ 2005.212477] Key type ._llcrypt registered [ 2005.220870] Key type .llcrypt registered [ 2005.498788] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2005.520696] alg: No test for adler32 (adler32-zlib) [ 2006.477491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2006.529029] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 2006.531934] LNet: *** cfs_fail_loc=e003, val=100*** [ 2007.342662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.3@tcp [ 2007.370381] LNet: *** cfs_fail_loc=e003, val=100*** [ 2007.374137] LNet: Skipped 2 previous similar messages [ 2007.383169] Ping reply received with mapping: [ 2007.386120] found metadata reply NI[192.168.204.3@tcp] [ 2007.390363] 03 cc a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2007.395253] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2007.399929] 5a 5a 5a 5a 5a 5a 5a 5a [ 2007.558185] LNet: 101566:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2007.565524] LNetError: 101566:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2007.590473] LNet: Removed LNI 192.168.204.3@tcp [ 2008.216087] Key type .llcrypt unregistered [ 2008.219835] Key type ._llcrypt unregistered [ 2019.271481] Key type ._llcrypt registered [ 2019.273911] Key type .llcrypt registered [ 2020.606978] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 19:27:16 (1790897236) [ 2021.256297] Key type .llcrypt unregistered [ 2021.259610] Key type ._llcrypt unregistered [ 2028.916796] Key type ._llcrypt registered [ 2028.920447] Key type .llcrypt registered [ 2029.125422] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2029.150167] alg: No test for adler32 (adler32-zlib) [ 2030.281502] LNetError: 102607:0:(api-ni.c:2855:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2030.748418] Key type .llcrypt unregistered [ 2030.750662] Key type ._llcrypt unregistered [ 2036.637927] Key type ._llcrypt registered [ 2036.643614] Key type .llcrypt registered [ 2038.299368] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 19:27:33 (1790897253) [ 2038.951227] Key type .llcrypt unregistered [ 2038.954524] Key type ._llcrypt unregistered [ 2045.445473] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2045.457414] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2048.239960] Key type ._llcrypt registered [ 2048.244244] Key type .llcrypt registered [ 2048.569283] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2048.606714] alg: No test for adler32 (adler32-zlib) [ 2049.703575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2049.782556] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2061.494421] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 19:27:56 (1790897276) [ 2061.807331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2061.826977] LNet: 104387:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2061.832352] LNetError: 104387:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2061.850219] LNet: Removed LNI 10.1.2.3@tcp [ 2061.868590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2062.659905] Key type .llcrypt unregistered [ 2062.665726] Key type ._llcrypt unregistered [ 2070.148165] Key type ._llcrypt registered [ 2070.153208] Key type .llcrypt registered [ 2070.330254] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2070.350183] alg: No test for adler32 (adler32-zlib) [ 2071.427678] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 2071.500264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.3@tcp [ 2071.760940] LNet: 104674:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2071.765395] LNetError: 104674:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2071.779701] LNet: Removed LNI 192.168.204.3@tcp [ 2072.626683] Key type .llcrypt unregistered [ 2072.631948] Key type ._llcrypt unregistered [ 2085.682866] Key type ._llcrypt registered [ 2085.686922] Key type .llcrypt registered [ 2087.364363] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 19:28:22 (1790897302) [ 2088.213489] Key type .llcrypt unregistered [ 2088.223955] Key type ._llcrypt unregistered [ 2095.555216] Key type ._llcrypt registered [ 2095.559354] Key type .llcrypt registered [ 2095.869512] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2095.893780] alg: No test for adler32 (adler32-zlib) [ 2103.442761] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 19:28:39 (1790897319) [ 2103.544240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2103.593108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2103.680617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2103.729312] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 2104.453670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2104.495206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2104.513214] LNet: Removed LNI 192.168.204.3@tcp [ 2104.795059] LNet: 106418:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2105.300769] Key type .llcrypt unregistered [ 2105.306161] Key type ._llcrypt unregistered [ 2117.025435] Key type ._llcrypt registered [ 2117.028923] Key type .llcrypt registered [ 2118.592335] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 19:28:53 (1790897333) [ 2119.042601] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2119.063632] alg: No test for adler32 (adler32-zlib) [ 2120.054694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2120.123358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2120.151733] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 2121.100195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.3@tcp [ 2121.148439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.204.3@tcp [ 2121.173358] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2121.180421] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2121.190445] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2121, 0, 0 [ 2121.204623] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2121.206529] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2121.212649] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2121.217523] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2121, 0, 0 [ 2121.223783] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2121.234648] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2121.237132] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2121.249078] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2121, 0, 2171 [ 2121.264561] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2121.274749] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2121.287837] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2121.293630] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2121, 0, 2171 [ 2121.307686] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2127.723905] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 19:29:03 (1790897343) [ 2128.160938] LNet: 107957:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2128.170502] LNetError: 107957:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2128.187807] LNet: Removed LNI 192.168.204.3@tcp [ 2128.978760] Key type .llcrypt unregistered [ 2128.980568] Key type ._llcrypt unregistered [ 2135.337369] Key type ._llcrypt registered [ 2135.340200] Key type .llcrypt registered [ 2135.610209] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2135.629873] alg: No test for adler32 (adler32-zlib) [ 2136.651349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2136.692664] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 2144.445969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.103@tcp [ 2154.808587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2154.882134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2154.914918] LNet: Added LNI 192.168.204.3@tcp1 [8/256/0/180] [ 2155.880750] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2156.084898] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2156.095295] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2156.105270] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2158.500903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2158.529713] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2170.970262] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 19:29:46 (1790897386) [ 2171.316500] LNet: 109463:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2171.331452] LNetError: 109463:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2171.344290] LNet: Removed LNI 192.168.204.3@tcp [ 2172.384379] LNet: Removed LNI 192.168.204.3@tcp1 [ 2172.389528] LNetError: 109463:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2172.400552] LNetError: 109463:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2172.420939] LNet: Removed LNI 10.1.2.3@tcp99 [ 2173.303980] Key type .llcrypt unregistered [ 2173.309641] Key type ._llcrypt unregistered [ 2180.370875] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2180.390450] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2182.044707] Key type ._llcrypt registered [ 2182.047555] Key type .llcrypt registered [ 2182.245394] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2182.264741] alg: No test for adler32 (adler32-zlib) [ 2183.343637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2183.409701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2183.447319] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 2184.441976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2184.464202] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2186.644548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2186.684750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2186.728288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2186.745305] LNetError: 110200:0:(peer.c:2366:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2186.768575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2186.812750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.3@tcp [ 2187.133383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2187.194430] LNet: 110220:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2187.202541] LNetError: 110220:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2187.208433] LNet: Removed LNI 10.1.2.3@tcp [ 2188.265467] LNet: Removed LNI 192.168.204.3@tcp [ 2189.155558] Key type .llcrypt unregistered [ 2189.159458] Key type ._llcrypt unregistered [ 2202.554936] Key type ._llcrypt registered [ 2202.561068] Key type .llcrypt registered [ 2204.145362] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 19:30:19 (1790897419) [ 2204.504734] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2204.522737] alg: No test for adler32 (adler32-zlib) [ 2205.501858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2205.604130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2205.630674] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 2206.587647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.3@tcp [ 2206.691866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg403-client@tcp [ 2213.650840] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 19:30:28 (1790897428) [ 2213.927331] LNet: 111754:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2213.941500] LNetError: 111754:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2213.968079] LNet: Removed LNI 192.168.204.3@tcp [ 2214.958593] Key type .llcrypt unregistered [ 2214.967966] Key type ._llcrypt unregistered [ 2221.748733] Key type ._llcrypt registered [ 2221.753223] Key type .llcrypt registered [ 2222.083686] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2222.125408] alg: No test for adler32 (adler32-zlib) [ 2223.059991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2223.091522] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 2223.132989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2230.095718] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 19:30:45 (1790897445) [ 2231.605594] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2233.384488] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 19:30:49 (1790897449) [ 2235.452063] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2237.520716] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 19:30:52 (1790897452) [ 2237.625421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2237.644737] LNet: 113171:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2237.652528] LNetError: 113171:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2237.670711] LNet: Removed LNI 192.168.204.3@tcp [ 2237.692141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2237.779967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2237.807373] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 2238.710184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2238.773559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2246.807355] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 19:31:02 (1790897462) [ 2247.045851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2247.072333] LNet: 113886:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2247.078543] LNetError: 113886:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2247.109330] LNet: Removed LNI 192.168.204.3@tcp [ 2247.145086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2247.234245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2247.268423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2247.924452] Key type .llcrypt unregistered [ 2247.933137] Key type ._llcrypt unregistered [ 2260.747614] Key type ._llcrypt registered [ 2260.749767] Key type .llcrypt registered [ 2262.595483] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 19:31:17 (1790897477) [ 2263.076443] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2263.092672] alg: No test for adler32 (adler32-zlib) [ 2264.097102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2264.181236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2264.236779] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 2265.077252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2265.170571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.3@tcp [ 2272.858554] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 19:31:28 (1790897488) [ 2273.126868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2273.201353] LNet: 115441:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2273.208657] LNetError: 115441:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2273.222444] LNet: Removed LNI 192.168.204.3@tcp [ 2273.250615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2273.290751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2273.313982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2280.621460] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 19:31:36 (1790897496) [ 2280.874817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2280.914704] LNet: 116021:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2280.950398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2281.076440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2281.368268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2281.393605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2281.456491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2281.842244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2281.886341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2281.976684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2282.012721] LNet: 116054:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2282.027947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2282.106328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2282.695303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2282.770174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2282.895715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2283.453048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2283.498754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2290.789584] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 19:31:46 (1790897506) [ 2291.072464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2291.096717] LNet: 116655:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2291.115403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2291.165486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2291.190208] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 2291.805885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2291.828399] LNet: Added LNI 192.168.204.3@tcp2 [8/256/0/180] [ 2292.924915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2293.005915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.3@tcp [ 2293.291321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2293.316575] LNetError: 116934:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2294.368732] LNet: Removed LNI 192.168.204.3@tcp [ 2294.376573] LNetError: 116934:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2294.406209] LNet: Removed LNI 192.168.204.3@tcp2 [ 2294.475694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2294.538597] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 2294.547977] LNet: Added LNI 192.168.204.3@tcp2 [8/256/0/180] [ 2294.554806] LNet: 116940:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2295.341315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.3@tcp [ 2295.594496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2295.627617] LNet: 116977:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2295.633598] LNet: 116977:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2296.673076] LNet: Removed LNI 192.168.204.3@tcp [ 2296.675099] LNetError: 116977:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2296.703697] LNetError: 116977:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2296.736694] LNet: Removed LNI 192.168.204.3@tcp2 [ 2296.841692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2296.981696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2297.039467] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 2298.355804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2298.409890] LNet: Added LNI 192.168.204.3@tcp2 [8/256/0/180] [ 2299.782776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.3@tcp [ 2299.926066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2300.021441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.3@tcp [ 2300.380348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2301.472165] LNet: Removed LNI 192.168.204.3@tcp [ 2301.490223] LNetError: 117262:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2301.518553] LNetError: 117262:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2301.591407] LNet: Removed LNI 192.168.204.3@tcp2 [ 2301.652500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2301.758971] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 2301.778855] LNet: Added LNI 192.168.204.3@tcp2 [8/256/0/180] [ 2301.795395] LNet: 117268:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2302.417726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.3@tcp [ 2302.689697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2302.724048] LNet: Removed LNI 192.168.204.3@tcp [ 2303.787680] LNet: Removed LNI 192.168.204.3@tcp2 [ 2303.871423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2304.027032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2304.087869] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 2305.086256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2305.110575] LNet: Added LNI 192.168.204.3@tcp2 [8/256/0/180] [ 2306.057037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2306.152748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.3@tcp [ 2306.426084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2306.450795] LNet: 117585:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2306.464428] LNet: 117585:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2307.488811] LNet: Removed LNI 192.168.204.3@tcp [ 2307.523137] LNet: Removed LNI 192.168.204.3@tcp2 [ 2307.578287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2307.688651] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 2307.708723] LNet: Added LNI 192.168.204.3@tcp2 [8/256/0/180] [ 2307.719993] LNet: 117591:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2308.493534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.3@tcp [ 2308.895639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2309.984459] LNet: Removed LNI 192.168.204.3@tcp [ 2309.991188] LNetError: 117628:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2310.006802] LNetError: 117628:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2310.027839] LNet: Removed LNI 192.168.204.3@tcp2 [ 2310.089071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2310.179245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2310.243760] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 2311.601497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2311.639403] LNet: Added LNI 192.168.204.3@tcp2 [8/256/0/180] [ 2313.526698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.3@tcp [ 2313.578721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2313.684176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.3@tcp [ 2314.066793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2315.168175] LNet: Removed LNI 192.168.204.3@tcp [ 2316.212670] LNet: Removed LNI 192.168.204.3@tcp2 [ 2316.296062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2316.354371] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 2316.369911] LNet: Added LNI 192.168.204.3@tcp2 [8/256/0/180] [ 2316.390437] LNet: 117919:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2317.333075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.3@tcp [ 2317.720812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2317.790626] LNet: Removed LNI 192.168.204.3@tcp [ 2318.841112] LNet: Removed LNI 192.168.204.3@tcp2 [ 2318.963898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2319.091041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2319.158946] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 2320.531684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2320.571857] LNet: Added LNI 192.168.204.3@tcp2 [8/256/0/180] [ 2321.855618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2321.927044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2322.060344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.3@tcp [ 2322.584478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2322.629725] LNet: 118253:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2322.642917] LNet: 118253:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2323.680722] LNet: Removed LNI 192.168.204.3@tcp [ 2323.708324] LNet: Removed LNI 192.168.204.3@tcp2 [ 2323.804446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2323.917751] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 2323.925626] LNet: Added LNI 192.168.204.3@tcp2 [8/256/0/180] [ 2323.935038] LNet: 118259:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2324.780804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.3@tcp [ 2325.367784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2326.432414] LNet: Removed LNI 192.168.204.3@tcp [ 2326.437785] LNetError: 118313:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2326.447868] LNetError: 118313:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2326.472469] LNet: Removed LNI 192.168.204.3@tcp2 [ 2326.534954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2326.673784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2326.721740] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 2328.242438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2328.306613] LNet: Added LNI 192.168.204.3@tcp2 [8/256/0/180] [ 2330.023569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.3@tcp [ 2330.111115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2330.179959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2330.305149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.3@tcp [ 2330.901568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2331.936268] LNet: Removed LNI 192.168.204.3@tcp [ 2332.967250] LNet: Removed LNI 192.168.204.3@tcp2 [ 2333.035592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2333.166550] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 2333.177026] LNet: Added LNI 192.168.204.3@tcp2 [8/256/0/180] [ 2333.194273] LNet: 118621:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2333.984818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.3@tcp [ 2341.984297] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2343.781491] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 19:32:39 (1790897559) [ 2343.922877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2343.951033] LNet: Removed LNI 192.168.204.3@tcp [ 2345.002632] LNet: Removed LNI 192.168.204.3@tcp2 [ 2345.064981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2345.418920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2345.447946] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 2346.488396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2346.516260] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2347.700288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2347.737573] LNet: Removed LNI 10.1.2.3@tcp [ 2347.748249] LNet: Removed LNI 192.168.204.3@tcp [ 2347.784248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2347.884956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2347.917246] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 2347.922057] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2348.364268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2348.401777] LNet: Removed LNI 192.168.204.3@tcp [ 2348.410782] LNet: Removed LNI 10.1.2.3@tcp [ 2354.512295] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 19:32:50 (1790897570) [ 2354.703974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2354.749525] LNet: 120248:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2354.752501] LNet: 120248:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2354.785148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2354.867312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2354.914974] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 2355.790931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2355.835645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2355.896390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.3@tcp [ 2355.951720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.3@tcp [ 2356.034502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.3@tcp [ 2356.076866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.3@tcp [ 2356.133454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.3@tcp [ 2356.196471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.3@tcp [ 2356.248466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.3@tcp [ 2356.294858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.3@tcp [ 2356.350434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.3@tcp [ 2356.417061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.3@tcp [ 2356.627422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2364.716635] Lustre: DEBUG MARKER: == sanity-lnet test 476: lnetctl set latency_stats toggles capture ========================================================== 19:33:00 (1790897580) [ 2364.958314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2365.022241] LNetError: 120992:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2365.034528] LNetError: 120992:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 2365.101812] LNet: Removed LNI 192.168.204.3@tcp [ 2365.151340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2365.258809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2365.288643] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 2366.433113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 0 [ 2366.465854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2366.509938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.3@tcp [ 2366.569450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.3@tcp [ 2366.651083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.3@tcp [ 2366.776291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.3@tcp [ 2366.838103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.3@tcp [ 2366.907621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.3@tcp [ 2366.981298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.3@tcp [ 2367.054624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.3@tcp [ 2367.120679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.3@tcp [ 2367.230367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.3@tcp [ 2367.371245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2367.444481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.3@tcp [ 2367.476232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.3@tcp [ 2367.539632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.3@tcp [ 2367.584097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.3@tcp [ 2367.627937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.3@tcp [ 2367.669893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.3@tcp [ 2367.730586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.3@tcp [ 2367.760262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.3@tcp [ 2367.788153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.3@tcp [ 2367.816864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.3@tcp [ 2377.063840] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 19:33:11 (1790897591) [ 2377.220440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2378.284393] LNet: Removed LNI 192.168.204.3@tcp [ 2378.341711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2378.417449] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2378.683115] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2378.691121] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2378.698316] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2381.084425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2381.103060] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 2382.032762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2382.083717] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2386.197981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.3@tcp [ 2386.291701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2386.451422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.204.3@tcp -m PUT -e local_timeout -r 1 [ 2386.614662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2386.785913] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2386.806906] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2387.329680] LNetError: 121815:0:(lib-move.c:4794:lnet_parse()) 192.168.204.3@tcp, src 192.168.204.3@tcp: Dropping PUT (error -108 looking up sender) [ 2388.386469] LNet: Removed LNI 10.1.2.3@tcp [ 2389.415773] LNet: Removed LNI 192.168.204.3@tcp [ 2390.125070] Key type .llcrypt unregistered [ 2390.127651] Key type ._llcrypt unregistered [ 2404.380738] Key type ._llcrypt registered [ 2404.382504] Key type .llcrypt registered [ 2406.454782] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 19:33:41 (1790897621) [ 2406.781622] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2406.795678] alg: No test for adler32 (adler32-zlib) [ 2408.011815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2408.826033] LNet: 123129:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2409.662560] Key type .llcrypt unregistered [ 2409.664875] Key type ._llcrypt unregistered [ 2417.961093] Key type ._llcrypt registered [ 2417.967069] Key type .llcrypt registered [ 2418.308992] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2418.331745] alg: No test for adler32 (adler32-zlib) [ 2419.293764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2419.339710] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 2429.202251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.103@tcp [ 2429.487240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2430.277852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.103@tcp [ 2430.625631] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.103@tcp [ 2430.644577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2430.697099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2430.746991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2430.800483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2430.858576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2430.923649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2431.006935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2431.074457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2431.194735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2431.263730] LNet: 123580:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2431.320726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2431.381813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2431.471948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2431.585784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2431.692390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2431.763090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2433.117335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2433.564463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2433.589900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2433.626691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2433.653580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2433.682648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2433.709861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2433.739237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2433.772355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2433.853711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2433.898142] LNet: 123700:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2433.934292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2433.998566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2434.054688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2434.156255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2434.234673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2434.312743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2435.014519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.103@tcp [ 2435.288562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2435.365566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2435.458756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2435.573867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2435.608273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2435.717558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2435.805632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2435.897111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2435.936902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2435.953167] LNet: 123819:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2435.963771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2435.985152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2436.019423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2436.083614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2436.175578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2436.265900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2437.469898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2437.976333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2438.030001] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2438.164516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2438.261420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2438.357571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2438.442360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2438.530930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2438.589444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2438.665358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2438.702272] LNet: 123938:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2438.737681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2438.795701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2438.834519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2438.886291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2438.928184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2438.963769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2439.465871] LNet: 123984:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2439.475290] LNetError: 123984:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2439.497033] LNet: Removed LNI 192.168.204.3@tcp [ 2440.387736] Key type .llcrypt unregistered [ 2440.397066] Key type ._llcrypt unregistered [ 2466.911556] Key type ._llcrypt registered [ 2466.916107] Key type .llcrypt registered [ 2468.754633] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 19:34:43 (1790897683) [ 2469.509187] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2469.542163] alg: No test for adler32 (adler32-zlib) [ 2470.486088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2470.981671] LNet: 124872:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2471.712799] Key type .llcrypt unregistered [ 2471.714661] Key type ._llcrypt unregistered [ 2479.594777] Key type ._llcrypt registered [ 2479.597705] Key type .llcrypt registered [ 2479.860836] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2479.877974] alg: No test for adler32 (adler32-zlib) [ 2480.838438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2480.881130] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 2488.806793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.103@tcp [ 2498.986324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2499.063632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2499.090394] LNet: Added LNI 192.168.204.3@tcp1 [8/256/0/180] [ 2500.075245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2500.659896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.103@tcp [ 2500.887268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2500.920451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2500.952126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2500.988835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2501.031945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2501.065903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2501.105252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2501.175326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2501.236686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2501.267476] LNet: 125527:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2501.285562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2501.331907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2501.392884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2501.410716] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.103@tcp [ 2501.447294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2501.487916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2501.530303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2502.285294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2502.642519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2502.688975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2502.728487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2502.759932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2502.796509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2502.844748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2502.885917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2502.926643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2502.958372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2502.972845] LNet: 125647:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2502.992910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2503.020562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2503.047444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2503.076911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2503.109782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2503.145105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2503.687914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.103@tcp [ 2503.919257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2503.948391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2503.978266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2504.014880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2504.052381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2504.082799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2504.110498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2504.149632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2504.196666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2504.219812] LNet: 125766:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2504.237993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2504.263800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2504.285626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2504.309598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2504.333489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2504.356827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2504.988327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2505.211487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2505.238485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2505.277403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2505.301303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2505.334144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2505.366741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2505.412305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2505.457602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2505.480945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2505.508284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2505.536978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2505.561729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2505.587866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2505.611839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2505.639150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2511.447699] LNet: 125966:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2511.456435] LNetError: 125966:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2511.461869] LNet: Removed LNI 192.168.204.3@tcp [ 2511.477520] LNet: Removed LNI 192.168.204.3@tcp1 [ 2511.852633] Key type .llcrypt unregistered [ 2511.854425] Key type ._llcrypt unregistered [ 2530.620747] Key type ._llcrypt registered [ 2530.623283] Key type .llcrypt registered [ 2532.218887] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 19:35:47 (1790897747) [ 2532.680753] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2532.728269] alg: No test for adler32 (adler32-zlib) [ 2533.702308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2533.785461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2533.841839] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 2534.799359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2534.834212] LNet: Added LNI 192.168.204.3@tcp2 [8/256/0/180] [ 2535.927920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.3@tcp [ 2536.035729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2537.056952] LNet: Removed LNI 192.168.204.3@tcp [ 2545.515777] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 19:36:00 (1790897760) [ 2548.170860] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2550.561444] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 19:36:06 (1790897766) [ 2561.049030] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 19:36:16 (1790897776) [ 2569.585782] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 19:36:25 (1790897785) [ 2578.595087] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 19:36:34 (1790897794) [ 2587.672720] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 19:36:43 (1790897803) [ 2596.485127] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 19:36:51 (1790897811) [ 2608.375987] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 19:37:03 (1790897823) [ 2618.033741] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 19:37:13 (1790897833) [ 2628.654295] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 19:37:24 (1790897844) [ 2636.709369] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 19:37:32 (1790897852) [ 2644.643422] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 19:37:40 (1790897860) [ 2651.998844] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 19:37:48 (1790897868) [ 2659.553922] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 19:37:55 (1790897875) [ 2669.446975] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 19:38:04 (1790897884) [ 2680.384654] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 19:38:15 (1790897895) [ 2689.418540] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 19:38:24 (1790897904) [ 2699.185821] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 19:38:34 (1790897914) [ 2707.299877] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 19:38:42 (1790897922) [ 2707.436051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2707.483701] LNet: 137662:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2707.508672] LNetError: 137662:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2707.567056] LNet: Removed LNI 192.168.204.3@tcp2 [ 2707.681272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2707.860915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2707.899383] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 2709.928491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2709.986234] LNet: 137809:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2709.995371] LNetError: 137809:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2710.027057] LNet: Removed LNI 192.168.204.3@tcp [ 2710.053064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2710.162610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2710.206807] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 2717.986366] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 19:38:53 (1790897933) [ 2726.786139] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 19:39:02 (1790897942) [ 2734.742437] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 19:39:10 (1790897950) [ 2743.016631] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 19:39:18 (1790897958) [ 2751.020512] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 19:39:26 (1790897966) [ 2759.787698] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 19:39:35 (1790897975) [ 2769.497613] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 19:39:44 (1790897984) [ 2777.987637] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 19:39:53 (1790897993) [ 2785.816206] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 19:40:01 (1790898001) [ 2794.666575] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 19:40:09 (1790898009) [ 2801.725282] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 19:40:17 (1790898017) [ 2809.504578] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 19:40:25 (1790898025) [ 2820.129599] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2577 sec ========== 19:40:34 (1790898034) [ 2821.053393] LNet: 145279:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2821.062285] LNetError: 145279:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2821.081265] LNet: Removed LNI 192.168.204.3@tcp [ 2822.307081] Key type .llcrypt unregistered [ 2822.311558] Key type ._llcrypt unregistered [ 2839.308808] Key type ._llcrypt registered [ 2839.312699] Key type .llcrypt registered [ 2839.901305] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2839.918897] alg: No test for adler32 (adler32-zlib) [ 2840.942887] Lustre: Lustre: Build Version: 2.17.58_63_g3705aed [ 2841.206618] LNet: Added LNI 192.168.204.3@tcp [8/256/0/180] [ 2842.952294] Key type lgssc registered [ 2844.519245] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2915.333249] Lustre: Mounted lustre-client - version 2.17.58_63_g3705aed [ 2921.175320] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2929.795249] Lustre: Unmounted lustre-client [ 2959.784440] Key type lgssc unregistered [ 2960.072806] LNet: 148240:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2960.083222] LNetError: 148240:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2960.110228] LNet: Removed LNI 192.168.204.3@tcp [ 2961.044343] Key type .llcrypt unregistered [ 2961.047609] Key type ._llcrypt unregistered