[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 437271014 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003252] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005013] kvm-guest: setup PV IPIs [ 0.008488] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010014] pid_max: default: 32768 minimum: 301 [ 0.011137] LSM: Security Framework initializing [ 0.012048] Yama: becoming mindful. [ 0.013036] SELinux: Initializing. [ 0.014076] *** VALIDATE selinux *** [ 0.022313] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027277] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028148] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029116] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030111] *** VALIDATE tmpfs *** [ 0.032468] *** VALIDATE proc *** [ 0.034054] *** VALIDATE cgroup *** [ 0.035008] *** VALIDATE cgroup2 *** [ 0.037112] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038128] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040030] Spectre V2 : User space: Vulnerable [ 0.041006] Speculative Store Bypass: Vulnerable [ 0.043916] debug: unmapping init [mem 0xffffffffa1e59000-0xffffffffa1e60fff] [ 0.046138] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047540] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048020] ... version: 2 [ 0.049010] ... bit width: 48 [ 0.050008] ... generic registers: 4 [ 0.051007] ... value mask: 0000ffffffffffff [ 0.052011] ... max period: 00007fffffffffff [ 0.053011] ... fixed-purpose events: 3 [ 0.054008] ... event mask: 000000070000000f [ 0.055341] rcu: Hierarchical SRCU implementation. [ 0.057471] smp: Bringing up secondary CPUs ... [ 0.058585] x86: Booting SMP configuration: [ 0.059021] .... node #0, CPUs: #1 #2 #3 [ 0.065025] smp: Brought up 1 node, 4 CPUs [ 0.067024] smpboot: Max logical packages: 1 [ 0.068012] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.135030] node 0 deferred pages initialised in 65ms [ 0.139037] devtmpfs: initialized [ 0.140164] x86/mm: Memory block size: 128MB [ 0.143743] gcov: version magic: 0x41383552 [ 0.146047] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.149110] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.152328] pinctrl core: initialized pinctrl subsystem [ 0.154184] [ 0.154812] ************************************************************* [ 0.156017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.158013] ** ** [ 0.160014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.164016] ** ** [ 0.166012] ** This means that this kernel is built to expose internal ** [ 0.168011] ** IOMMU data structures, which may compromise security on ** [ 0.170012] ** your system. ** [ 0.172011] ** ** [ 0.174011] ** If you see this message and you are not debugging the ** [ 0.176017] ** kernel, report this immediately to your vendor! ** [ 0.178010] ** ** [ 0.180010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.182010] ************************************************************* [ 0.184754] NET: Registered protocol family 16 [ 0.186450] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.189055] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.191055] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.194426] cpuidle: using governor menu [ 0.196598] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.198397] PCI: Using configuration type 1 for base access [ 0.200135] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.208124] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.211078] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.214089] cryptd: max_cpu_qlen set to 1000 [ 0.216297] ACPI: Added _OSI(Module Device) [ 0.218016] ACPI: Added _OSI(Processor Device) [ 0.220012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.221011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.226547] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.232510] ACPI: Interpreter enabled [ 0.233055] ACPI: PM: (supports S0 S3 S4 S5) [ 0.235014] ACPI: Using IOAPIC for interrupt routing [ 0.237134] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.240522] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.251893] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.254041] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.257020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.260171] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.266489] acpiphp: Slot [2] registered [ 0.268118] acpiphp: Slot [5] registered [ 0.269109] acpiphp: Slot [6] registered [ 0.271102] acpiphp: Slot [3] registered [ 0.272143] acpiphp: Slot [4] registered [ 0.274102] acpiphp: Slot [7] registered [ 0.276074] acpiphp: Slot [8] registered [ 0.278170] acpiphp: Slot [9] registered [ 0.280161] acpiphp: Slot [10] registered [ 0.282097] acpiphp: Slot [11] registered [ 0.283108] acpiphp: Slot [12] registered [ 0.286153] acpiphp: Slot [13] registered [ 0.287089] acpiphp: Slot [14] registered [ 0.289091] acpiphp: Slot [15] registered [ 0.292114] acpiphp: Slot [16] registered [ 0.293104] acpiphp: Slot [17] registered [ 0.295083] acpiphp: Slot [18] registered [ 0.297103] acpiphp: Slot [19] registered [ 0.299101] acpiphp: Slot [20] registered [ 0.300084] acpiphp: Slot [21] registered [ 0.302086] acpiphp: Slot [22] registered [ 0.304155] acpiphp: Slot [23] registered [ 0.306108] acpiphp: Slot [24] registered [ 0.308086] acpiphp: Slot [25] registered [ 0.309123] acpiphp: Slot [26] registered [ 0.311092] acpiphp: Slot [27] registered [ 0.313086] acpiphp: Slot [28] registered [ 0.315085] acpiphp: Slot [29] registered [ 0.317089] acpiphp: Slot [30] registered [ 0.319091] acpiphp: Slot [31] registered [ 0.321065] PCI host bridge to bus 0000:00 [ 0.322017] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.324031] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.327019] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.330021] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.333019] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.336024] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.339163] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.342527] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.344622] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.352011] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.356058] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.359015] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.362017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.364015] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.367803] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.369711] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.372039] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.374498] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.378836] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.389018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.393015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.398049] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.405021] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.409016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.419012] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.428523] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.441015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.447012] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.467016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.475803] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.479100] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.486430] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.488450] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.490184] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.494172] iommu: Default domain type: Passthrough [ 0.495266] SCSI subsystem initialized [ 0.497161] ACPI: bus type USB registered [ 0.498141] usbcore: registered new interface driver usbfs [ 0.500075] usbcore: registered new interface driver hub [ 0.501068] usbcore: registered new device driver usb [ 0.503131] pps_core: LinuxPPS API ver. 1 registered [ 0.505010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.508063] PTP clock support registered [ 0.510312] EDAC MC: Ver: 3.0.0 [ 0.512147] PCI: Using ACPI for IRQ routing [ 0.513771] NetLabel: Initializing [ 0.515010] NetLabel: domain hash size = 128 [ 0.516006] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.518078] NetLabel: unlabeled traffic allowed by default [ 0.521093] vgaarb: loaded [ 0.522317] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.524012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.531000] clocksource: Switched to clocksource kvm-clock [ 0.648756] VFS: Disk quotas dquot_6.6.0 [ 0.650357] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.653114] *** VALIDATE ramfs *** [ 0.654248] *** VALIDATE hugetlbfs *** [ 0.655462] pnp: PnP ACPI init [ 0.657982] pnp: PnP ACPI: found 6 devices [ 0.680702] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.683291] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.684705] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.686159] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.687890] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.689937] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.692596] NET: Registered protocol family 2 [ 0.694912] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.699765] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.702993] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.707633] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.710608] TCP: Hash tables configured (established 65536 bind 65536) [ 0.713609] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.716778] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.719767] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.722247] NET: Registered protocol family 1 [ 0.724483] RPC: Registered named UNIX socket transport module. [ 0.725858] RPC: Registered udp transport module. [ 0.727478] RPC: Registered tcp transport module. [ 0.728799] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.730952] NET: Registered protocol family 44 [ 0.732593] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.734448] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.736465] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.738855] PCI: CLS 0 bytes, default 64 [ 0.740857] Unpacking initramfs... [ 2.052956] debug: unmapping init [mem 0xffff94877cc64000-0xffff94877ffcffff] [ 2.055632] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.056900] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.058513] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.522983] Initialise system trusted keyrings [ 2.525445] Key type blacklist registered [ 2.527323] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.542997] zbud: loaded [ 2.549167] *** VALIDATE nfs *** [ 2.550368] *** VALIDATE nfs4 *** [ 2.551944] pstore: using deflate compression [ 2.555037] Platform Keyring initialized [ 2.650451] NET: Registered protocol family 38 [ 2.652684] Key type asymmetric registered [ 2.654268] Asymmetric key parser 'x509' registered [ 2.656323] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.659722] io scheduler mq-deadline registered [ 2.661123] io scheduler kyber registered [ 2.662548] io scheduler bfq registered [ 2.664496] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.667399] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.670288] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.673339] ACPI: Power Button [PWRF] [ 2.678127] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.684149] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.700058] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.730447] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.760456] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.766353] Non-volatile memory driver v1.3 [ 2.768227] Linux agpgart interface v0.103 [ 2.798606] virtio_blk virtio1: [vda] 149760 512-byte logical blocks (76.7 MB/73.1 MiB) [ 2.801796] vda: detected capacity change from 0 to 76677120 [ 2.819383] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.822687] vdb: detected capacity change from 0 to 1073741824 [ 2.828860] libphy: Fixed MDIO Bus: probed [ 2.834132] usbcore: registered new interface driver usbserial_generic [ 2.837063] usbserial: USB Serial support registered for generic [ 2.839741] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.844197] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.846171] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.848986] mousedev: PS/2 mouse device common for all mice [ 2.852205] rtc_cmos 00:05: RTC can wake from S4 [ 2.857435] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.857651] rtc_cmos 00:05: registered as rtc0 [ 2.861287] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.863684] intel_pstate: CPU model not supported [ 2.866625] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.868067] hid: raw HID events driver (C) Jiri Kosina [ 2.872157] usbcore: registered new interface driver usbhid [ 2.873657] usbhid: USB HID core driver [ 2.873831] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.874744] drop_monitor: Initializing network drop monitor service [ 2.874926] Initializing XFRM netlink socket [ 2.884420] NET: Registered protocol family 10 [ 2.886913] Segment Routing with IPv6 [ 2.887896] NET: Registered protocol family 17 [ 2.889165] mpls_gso: MPLS GSO support [ 2.893145] RAS: Correctable Errors collector initialized. [ 2.895674] AVX version of gcm_enc/dec engaged. [ 2.897551] AES CTR mode by8 optimization enabled [ 2.964444] sched_clock: Marking stable (2964425792, 0)->(3870422029, -905996237) [ 2.966551] registered taskstats version 1 [ 2.967580] Loading compiled-in X.509 certificates [ 2.969214] zswap: loaded using pool lzo/zbud [ 2.987948] Key type big_key registered [ 2.997945] Key type encrypted registered [ 2.999454] ima: No TPM chip found, activating TPM-bypass! [ 3.001196] ima: Allocated hash algorithm: sha1 [ 3.003060] ima: No architecture policies found [ 3.004809] evm: Initialising EVM extended attributes: [ 3.006600] evm: security.selinux [ 3.007578] evm: security.ima [ 3.008681] evm: security.capability [ 3.009915] evm: HMAC attrs: 0x1 [ 3.012325] rtc_cmos 00:05: setting system clock to 2026-09-14 09:21:52 UTC (1789377712) [ 3.018766] debug: unmapping init [mem 0xffffffffa2e03000-0xffffffffa2ffffff] [ 3.021136] debug: unmapping init [mem 0xffffffffa1b82000-0xffffffffa1e58fff] [ 3.028059] Write protecting the kernel read-only data: 28672k [ 3.031973] debug: unmapping init [mem 0xffffffffa0203000-0xffffffffa03fffff] [ 3.034929] debug: unmapping init [mem 0xffffffffa0b14000-0xffffffffa0bfffff] [ 3.066591] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.076197] systemd[1]: Detected virtualization kvm. [ 3.079668] systemd[1]: Detected architecture x86-64. [ 3.081888] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.108367] systemd[1]: No hostname configured. [ 3.109473] systemd[1]: Set hostname to . [ 3.110982] random: systemd: uninitialized urandom read (16 bytes read) [ 3.112798] systemd[1]: Initializing machine ID from random generator. [ 3.259277] random: systemd: uninitialized urandom read (16 bytes read) [ 3.262228] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.266480] random: systemd: uninitialized urandom read (16 bytes read) [ 3.269076] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.274154] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Swap. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... Starting Setup Virtual Console... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Initrd Root Device. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.001855] device-mapper: uevent: version 1.0.3 [ 4.004211] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.712114] virtio_net virtio0 ens2: renamed from eth0 [ 4.719019] random: fast init done [ 4.742566] scsi host0: ata_piix [ 4.780579] scsi host1: ata_piix [ 4.782597] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.784913] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.555663] dracut-initqueue[574]: RTNETLINK answers: File exists [ 9.521773] random: crng init done [ 9.523107] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.048033] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.267364] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.561504] SELinux: Disabled at runtime. [ 11.620055] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.633341] systemd[1]: Detected virtualization kvm. [ 11.635576] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.188265] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.192092] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.199228] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.202841] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.206426] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.216606] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.242841] systemd[1]: Listening on Process Core Dump Socket. [ OK ] Listening on Process Core Dump Socket. Mounting POSIX Message Queue File System... [ OK ] Listening on udev Control Socket. Starting Apply Kernel Variables... [ OK ] Created slice User and Session Slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Kernel Debug File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target Slices. Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-f[ 12.388931] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS s-binfmt_misc.automount' for details. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Mounting Huge Pages File System... [ OK ] Reached target rpc_pipefs.target. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.866901] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.274616] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.279610] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.388377] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.416313] EDAC sbridge: Ver: 1.1.2 [ 14.869724] Key type dns_resolver registered [ 15.167138] NFS: Registering the id_resolver key type [ 15.169071] Key type id_resolver registered [ 15.170448] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg210-client login: [ 54.119765] libcfs: loading out-of-tree module taints kernel. [ 54.299421] Key type ._llcrypt registered [ 54.309726] Key type .llcrypt registered [ 55.303099] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 55.338151] alg: No test for adler32 (adler32-zlib) [ 57.075604] Lustre: Lustre: Build Version: 2.17.58_2_g65a210d [ 58.276199] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 60.175762] Key type lgssc registered [ 62.174875] Lustre: Echo OBD driver; http://www.lustre.org/ [ 136.481242] hrtimer: interrupt took 7231208 ns [ 256.724803] Lustre: Mounted lustre-client - version 2.17.58_2_g65a210d [ 262.523905] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 279.382196] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing check_logdir /tmp/testlogs/ [ 282.595140] Lustre: lustre-OST0000-osc-ffff9487c7f63000: disconnect after 23s idle [ 284.488420] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing yml_node [ 288.275507] Lustre: DEBUG MARKER: Client: 2.17.58.2 [ 290.860596] Lustre: DEBUG MARKER: MDS: 2.17.58.2 [ 293.350769] Lustre: DEBUG MARKER: OSS: 2.17.58.2 [ 294.512357] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Mon Sep 14 05:26:43 EDT 2026 [ 314.641877] Lustre: Unmounted lustre-client [ 369.168225] Key type lgssc unregistered [ 369.545540] LNet: 9226:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 369.559743] LNetError: 9226:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 369.608088] LNet: Removed LNI 192.168.202.10@tcp [ 370.548202] Key type .llcrypt unregistered [ 370.551724] Key type ._llcrypt unregistered [ 407.550532] Key type ._llcrypt registered [ 407.556258] Key type .llcrypt registered [ 407.760065] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 407.776112] alg: No test for adler32 (adler32-zlib) [ 408.699456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 408.829019] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 408.839176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 413.412458] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 413.559188] LNet: 10151:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 413.569571] LNetError: 10151:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 413.606073] LNet: Removed LNI 192.168.202.10@tcp [ 414.363503] Key type .llcrypt unregistered [ 414.365981] Key type ._llcrypt unregistered [ 421.885377] Key type ._llcrypt registered [ 421.888129] Key type .llcrypt registered [ 423.863270] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 05:28:51 (1789378131) [ 424.267895] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 424.297641] alg: No test for adler32 (adler32-zlib) [ 425.197221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 425.282304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 425.315736] LNet: 10570:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 425.332980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 425.392435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 425.406201] LNet: 10577:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 431.879629] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 05:29:00 (1789378140) [ 432.347498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 432.364994] LNet: 11164:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 432.398021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 432.448180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 439.897564] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 05:29:07 (1789378147) [ 440.130152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 440.187215] LNet: 11750:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 440.241137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 440.536877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 453.170956] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 05:29:19 (1789378159) [ 453.352720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 453.478662] LNet: 12346:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 453.537111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 453.758721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 464.145840] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 05:29:31 (1789378171) [ 464.228527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 464.248540] LNet: 12937:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 464.264848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 464.345788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 464.647591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 472.571923] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 05:29:40 (1789378180) [ 472.910664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 472.973316] LNet: 13543:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 473.013592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 473.109423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 473.738458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 474.532337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 483.265342] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 05:29:51 (1789378191) [ 483.593545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 483.618437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 483.690694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 493.737955] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 05:30:01 (1789378201) [ 493.905074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 493.948773] LNet: 14733:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 493.961752] LNet: 14733:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 494.027867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 494.187687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 494.219203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 495.025259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 495.082695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 495.879792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 495.958447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 496.711323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 496.828683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 497.578553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 497.615565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 498.262869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 498.290907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 498.670363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 498.758284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 499.509295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 499.629371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 500.421181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 500.464413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 501.037444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 501.103616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 501.795900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 501.958564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 502.692702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 502.774727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 503.725353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 503.813636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 514.199225] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 05:30:21 (1789378221) [ 514.388316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 514.450746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 514.631987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 514.669428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 526.892411] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 05:30:32 (1789378232) [ 527.111760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 527.172044] LNet: 16022:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 527.180410] LNet: 16022:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 527.205461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 527.442644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 527.487188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 537.368708] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 05:30:45 (1789378245) [ 537.571652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 537.676479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 537.782853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 537.821629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 545.671151] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 05:30:53 (1789378253) [ 545.872767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 545.971704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 546.163730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 546.261532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 559.079876] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 05:31:06 (1789378266) [ 559.651552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 559.725855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 559.976907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 560.052616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 573.918148] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 05:31:21 (1789378281) [ 574.147356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 574.228518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 574.451674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 574.560702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 584.082635] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 05:31:31 (1789378291) [ 584.411824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 584.436951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 584.547389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 585.000685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 592.739816] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 05:31:40 (1789378300) [ 592.846067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 592.894571] LNet: 19599:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 592.899226] LNet: 19599:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 592.954679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 593.296969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 593.704278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 604.604287] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 05:31:51 (1789378311) [ 604.890073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 604.993850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 605.351131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 605.923354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 617.855570] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 05:32:05 (1789378325) [ 618.005055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 618.066178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 618.546809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 618.840576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 626.804291] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 05:32:14 (1789378334) [ 627.089518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 627.142545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 627.238652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 627.336727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 635.649955] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 05:32:23 (1789378343) [ 635.943626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 636.027186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 636.132831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 636.180802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 644.220605] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 05:32:31 (1789378351) [ 644.339542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 644.395918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 644.525274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 652.681109] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 05:32:40 (1789378360) [ 652.797502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 652.863196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 652.948939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 660.460977] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 05:32:48 (1789378368) [ 660.657815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 660.732841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 660.921111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 661.590597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 669.759965] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 05:32:57 (1789378377) [ 670.010804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 670.069218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 670.184789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 670.247320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 678.308224] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 05:33:06 (1789378386) [ 678.484339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 678.529138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 678.593122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 678.636197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 685.275774] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 05:33:13 (1789378393) [ 685.424783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 685.514750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 685.653420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 685.707569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 694.494673] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 05:33:22 (1789378402) [ 694.626136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 694.711553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 694.842742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 694.893770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 703.803262] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 05:33:31 (1789378411) [ 703.930136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 703.991579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 704.055573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 704.151559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 704.168940] LNetError: 26908:0:(peer.c:2366:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 704.224706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 704.253269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 711.871698] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 05:33:39 (1789378419) [ 712.274325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 712.326899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 712.423566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.vDWKRXZmnI [ 712.555738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.vDWKRXZmnI [ 712.654141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.vDWKRXZmnI [ 721.392929] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 05:33:48 (1789378428) [ 721.449614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 721.486156] LNet: 28090:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 721.495930] LNet: 28090:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 13 previous similar messages [ 721.535712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 721.589345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 721.624651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 721.676152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 732.339150] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 05:33:59 (1789378439) [ 732.461404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 732.570445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 732.839912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 732.930111] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 735.085812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 737.735199] Lustre: Lustre: Build Version: 2.17.58_2_g65a210d [ 739.871336] Key type lgssc registered [ 741.660325] Lustre: Echo OBD driver; http://www.lustre.org/ [ 748.374426] LNetError: 28695:0:(lib-move.c:2151:lnet_handle_find_routed_path()) peer 192.168.202.10@tcp99 has no available nets [ 753.116405] Key type lgssc unregistered [ 753.543920] LNetError: 29829:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 754.600195] LNet: Removed LNI 192.168.202.10@tcp [ 755.403118] Key type .llcrypt unregistered [ 755.413741] Key type ._llcrypt unregistered [ 766.019503] Key type ._llcrypt registered [ 766.027809] Key type .llcrypt registered [ 767.435853] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 05:34:35 (1789378475) [ 768.327455] Key type .llcrypt unregistered [ 768.336704] Key type ._llcrypt unregistered [ 776.035430] Key type ._llcrypt registered [ 776.040621] Key type .llcrypt registered [ 776.218584] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 776.227910] alg: No test for adler32 (adler32-zlib) [ 777.284870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 777.420360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 777.443137] LNet: 30750:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 777.448923] LNet: 30750:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 777.468316] LNet: 30750:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 777.494968] LNet: Message forwarding will be enabled in 2s [ 779.551094] LNet: Message forwarding enabled [ 780.088384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 780.152480] LNet: Message forwarding will stop in 2s to 6s [ 783.668878] LNet: Message forwarding disabled [ 783.863828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 783.908157] LNet: 30829:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 783.924529] LNet: 30829:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 783.940899] LNet: 30829:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 783.984154] LNet: Message forwarding will be enabled in 2s [ 784.282187] LNet: 30842:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 785.346501] Key type .llcrypt unregistered [ 785.351868] Key type ._llcrypt unregistered [ 803.471799] Key type ._llcrypt registered [ 803.473312] Key type .llcrypt registered [ 804.974529] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 05:35:12 (1789378512) [ 805.544972] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 805.603145] alg: No test for adler32 (adler32-zlib) [ 806.524677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 806.596530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 817.293904] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 05:35:24 (1789378524) [ 817.441352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 817.459429] LNet: 32262:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 817.476723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 817.564925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 817.594624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 817.613372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 817.630926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 817.659872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 817.683791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 817.700630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 817.728707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 817.748339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 817.774744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 817.793834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 817.813729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 817.843265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 817.875889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 817.901933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 817.955721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 818.047504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 818.165956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 818.237385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 818.280977] LNetError: 32306:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 827.251084] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 05:35:34 (1789378534) [ 827.456568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 827.515343] LNet: 32878:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 827.575893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 827.733354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 837.044747] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 05:35:44 (1789378544) [ 839.422806] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg210-server [ 841.766799] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 05:35:49 (1789378549) [ 841.965380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 842.017137] LNet: 33830:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 842.073984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 842.131844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 842.155812] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 843.244909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.202.[11-15]@tcp [ 843.267502] LNet: 33965:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 843.300239] LNetError: 33968:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp1 through 192.168.202.15@tcp has gone from up to down [ 843.352917] LNetError: 33835:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.13@tcp is being used as a gateway but routing feature is not turned on [ 851.990430] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 05:35:59 (1789378559) [ 854.059910] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg210-server [ 855.778064] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 05:36:03 (1789378563) [ 855.989598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 856.014544] LNet: 34916:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 856.022451] LNetError: 34916:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 856.068935] LNet: Removed LNI 192.168.202.10@tcp [ 856.098483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 856.170991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 856.209373] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 857.001579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.202.[11-16/2]@tcp [ 857.034885] LNet: 35033:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 857.049131] LNet: 35033:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 857.067278] LNetError: 34920:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.11@tcp is being used as a gateway but routing feature is not turned on [ 857.080470] LNetError: 34920:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 7 previous similar messages [ 857.081293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.202.[11-16/2]@tcp [ 857.094481] LNetError: 34920:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp103 through 192.168.202.11@tcp has gone from up to down [ 857.128832] LNetError: 34920:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) Skipped 5 previous similar messages [ 865.103727] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 05:36:12 (1789378572) [ 865.213186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 865.248332] LNet: 35625:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 865.259962] LNetError: 35625:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 865.294111] LNet: Removed LNI 192.168.202.10@tcp [ 865.380640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 865.512664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 865.612853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 865.684913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 865.743382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 865.794471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 865.828218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 865.886654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 865.979180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 866.008708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 866.065912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 866.145504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 866.184953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 866.241974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 866.327142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 866.357966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 866.419556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 866.505245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 866.537890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 866.581231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 866.665518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 866.721637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 866.788395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 866.872961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 866.904249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 866.984482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 867.098389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 867.179561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 867.287975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 867.429947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 867.490689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 867.581445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 867.711855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 867.815583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 867.925571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 875.314786] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 05:36:22 (1789378582) [ 877.208338] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg210-server [ 879.120272] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 05:36:27 (1789378587) [ 880.868463] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg210-server [ 882.528488] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 05:36:30 (1789378590) [ 882.637354] LNet: 37045:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 882.647328] LNet: 37045:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 883.405109] Key type .llcrypt unregistered [ 883.414198] Key type ._llcrypt unregistered [ 890.229204] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 890.378780] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 890.385214] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 890.396484] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 893.248849] Key type ._llcrypt registered [ 893.257214] Key type .llcrypt registered [ 893.463507] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 893.488885] alg: No test for adler32 (adler32-zlib) [ 894.437871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 894.480786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 894.498092] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 895.140165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 895.155715] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 896.660481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 896.674571] LNet: Removed LNI 10.1.2.3@tcp [ 896.765438] LNet: 37773:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 896.769538] LNetError: 37773:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 896.782690] LNet: Removed LNI 192.168.202.10@tcp [ 897.293371] Key type .llcrypt unregistered [ 897.295094] Key type ._llcrypt unregistered [ 909.533108] Key type ._llcrypt registered [ 909.534620] Key type .llcrypt registered [ 911.070982] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 05:36:58 (1789378618) [ 911.804482] Key type .llcrypt unregistered [ 911.806931] Key type ._llcrypt unregistered [ 919.910770] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 920.118661] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 920.130583] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 920.141848] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 923.138766] Key type ._llcrypt registered [ 923.140617] Key type .llcrypt registered [ 923.630371] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 923.673050] alg: No test for adler32 (adler32-zlib) [ 924.552794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 924.628938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 924.676631] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 925.671951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 925.688921] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 928.597137] LNet: 39275:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 928.604645] LNetError: 39275:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 928.616995] LNet: Removed LNI 10.1.2.3@tcp [ 928.625701] LNet: Removed LNI 192.168.202.10@tcp [ 929.229405] Key type .llcrypt unregistered [ 929.231756] Key type ._llcrypt unregistered [ 946.421522] Key type ._llcrypt registered [ 946.424251] Key type .llcrypt registered [ 948.397996] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 05:37:35 (1789378655) [ 949.344405] Key type .llcrypt unregistered [ 949.347945] Key type ._llcrypt unregistered [ 957.085046] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 957.236880] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 957.239347] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 957.242352] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 959.865419] Key type ._llcrypt registered [ 959.868425] Key type .llcrypt registered [ 960.108188] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 960.127246] alg: No test for adler32 (adler32-zlib) [ 961.024485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 961.079417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 961.100331] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 962.930776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 962.968495] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 964.786548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 964.802575] LNet: Removed LNI 10.1.2.3@tcp [ 964.811478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 964.836703] LNet: Removed LNI 10.1.2.31@tcp [ 964.866815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 964.895751] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 965.722453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 965.745816] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 966.327946] LNet: 41196:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 966.331171] LNetError: 41196:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 966.333918] LNet: Removed LNI 10.1.2.31@tcp [ 966.353037] LNet: Removed LNI 10.1.2.3@tcp [ 966.698826] Key type .llcrypt unregistered [ 966.705157] Key type ._llcrypt unregistered [ 977.961590] Key type ._llcrypt registered [ 977.964206] Key type .llcrypt registered [ 979.499228] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 05:38:07 (1789378687) [ 979.931516] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 979.939954] alg: No test for adler32 (adler32-zlib) [ 981.002321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 981.147026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 981.205397] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 982.336780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 982.365719] LNet: 42155:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 982.369965] LNetError: 42155:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 982.397323] LNet: Removed LNI 192.168.202.10@tcp [ 982.424749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 982.496669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.202.10@tcp [ 982.517672] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 983.252514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.202.10@tcp [ 983.290779] LNet: Removed LNI 192.168.202.10@tcp [ 983.389242] LNet: 42290:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 984.483404] Key type .llcrypt unregistered [ 984.485072] Key type ._llcrypt unregistered [ 1002.674893] Key type ._llcrypt registered [ 1002.678992] Key type .llcrypt registered [ 1004.715536] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 05:38:31 (1789378711) [ 1007.059468] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg210-server [ 1009.949271] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 05:38:37 (1789378717) [ 1010.958308] Key type .llcrypt unregistered [ 1010.963254] Key type ._llcrypt unregistered [ 1018.294706] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1018.425891] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1018.435847] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1018.447778] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1020.841858] Key type ._llcrypt registered [ 1020.843982] Key type .llcrypt registered [ 1020.990972] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1021.002875] alg: No test for adler32 (adler32-zlib) [ 1021.854220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1021.895977] LNet: Added LNI 192.168.202.10@tcp1 [8/256/0/180] [ 1022.443058] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 1024.452711] LNet: 44140:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1024.459914] LNetError: 44140:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1024.466682] LNet: Removed LNI 192.168.202.10@tcp1 [ 1024.484643] LNet: Removed LNI 10.1.2.3@tcp2 [ 1025.168714] Key type .llcrypt unregistered [ 1025.174105] Key type ._llcrypt unregistered [ 1037.495286] Key type ._llcrypt registered [ 1037.499719] Key type .llcrypt registered [ 1039.508122] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 05:39:06 (1789378746) [ 1039.737428] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1039.745143] alg: No test for adler32 (adler32-zlib) [ 1040.683898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1040.717874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1040.733898] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1041.310413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1041.325754] LNet: 45118:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1041.331908] LNetError: 45118:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1041.348693] LNet: Removed LNI 192.168.202.10@tcp [ 1041.360210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1041.419242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 1041.437489] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1049.088305] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 05:39:17 (1789378757) [ 1049.468312] LNet: 45717:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1049.475410] LNetError: 45717:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1049.508823] LNet: Removed LNI 192.168.202.10@tcp [ 1049.941570] Key type .llcrypt unregistered [ 1049.943148] Key type ._llcrypt unregistered [ 1057.191670] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1057.305530] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1057.308502] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1057.311495] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1060.046989] Key type ._llcrypt registered [ 1060.051683] Key type .llcrypt registered [ 1060.369651] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1060.402745] alg: No test for adler32 (adler32-zlib) [ 1061.495761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1061.606065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1061.674883] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1062.552379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1062.584630] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1065.103420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1065.144513] LNet: 46459:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1065.158786] LNetError: 46459:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1065.184383] LNet: Removed LNI 10.1.2.3@tcp [ 1065.222346] LNet: Removed LNI 192.168.202.10@tcp [ 1065.274625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1065.376704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 1065.401335] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1065.407325] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1065.973536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1065.987697] LNet: 46487:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1065.992685] LNetError: 46487:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1065.997493] LNetError: 46487:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1066.005355] LNet: Removed LNI 10.1.2.3@tcp [ 1066.012764] LNet: Removed LNI 192.168.202.10@tcp [ 1066.029818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1066.102245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 1066.124870] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1066.128331] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1066.363333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1066.376517] LNet: Added LNI 192.168.202.10@tcp2 [8/256/0/180] [ 1067.136696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1067.155382] LNet: 46649:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1067.158634] LNetError: 46649:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1067.162359] LNetError: 46649:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1067.167100] LNet: Removed LNI 10.1.2.3@tcp [ 1067.168917] LNet: Removed LNI 192.168.202.10@tcp [ 1067.178742] LNet: Removed LNI 192.168.202.10@tcp2 [ 1067.191233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1067.264918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 1067.282257] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1067.290280] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1067.295895] LNet: Added LNI 192.168.202.10@tcp2 [8/256/0/180] [ 1075.368752] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 05:39:43 (1789378783) [ 1075.593361] LNet: 47254:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1075.599883] LNetError: 47254:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1075.604605] LNetError: 47254:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1075.612114] LNet: Removed LNI 10.1.2.3@tcp [ 1075.613981] LNet: Removed LNI 192.168.202.10@tcp [ 1075.625911] LNet: Removed LNI 192.168.202.10@tcp2 [ 1076.078962] Key type .llcrypt unregistered [ 1076.083402] Key type ._llcrypt unregistered [ 1083.265672] Key type ._llcrypt registered [ 1083.271793] Key type .llcrypt registered [ 1083.515618] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1083.529410] alg: No test for adler32 (adler32-zlib) [ 1084.537122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1084.580585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1084.600774] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1085.743691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1085.760819] LNet: 47648:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1085.765773] LNetError: 47648:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1085.783923] LNet: Removed LNI 192.168.202.10@tcp [ 1085.814918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1085.888697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 1085.934110] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1086.511559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1086.529690] LNet: 47678:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1086.537100] LNetError: 47678:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1086.552791] LNet: Removed LNI 192.168.202.10@tcp [ 1086.592507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1086.707782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 1086.737333] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1087.427615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1087.457173] LNet: Removed LNI 192.168.202.10@tcp [ 1087.492625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1087.588937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 1087.607944] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1088.313695] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1088.506184] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1088.517254] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1088.522737] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1091.102768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1091.131279] LNet: 47787:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1091.137605] LNet: 47787:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1091.148297] LNetError: 47787:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1091.155204] LNetError: 47787:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1091.169909] LNet: Removed LNI 192.168.202.10@tcp [ 1091.193937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1091.295249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 1091.356793] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1099.260503] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 05:40:07 (1789378807) [ 1099.402923] LNet: 48385:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1099.414587] LNetError: 48385:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1099.445603] LNet: Removed LNI 192.168.202.10@tcp [ 1100.142141] Key type .llcrypt unregistered [ 1100.144320] Key type ._llcrypt unregistered [ 1106.148253] Key type ._llcrypt registered [ 1106.152522] Key type .llcrypt registered [ 1106.357995] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1106.387746] alg: No test for adler32 (adler32-zlib) [ 1107.378439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1107.432089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1107.450350] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1108.293253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1108.305907] LNet: 48790:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1108.310299] LNetError: 48790:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1108.318686] LNet: Removed LNI 192.168.202.10@tcp [ 1108.330662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1108.379646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 1108.402742] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1114.599614] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 05:40:22 (1789378822) [ 1114.690317] LNet: 49383:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1114.705895] LNetError: 49383:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1114.729229] LNet: Removed LNI 192.168.202.10@tcp [ 1115.428655] Key type .llcrypt unregistered [ 1115.432364] Key type ._llcrypt unregistered [ 1121.669627] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1121.853970] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1121.860545] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1121.873825] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1123.630494] Key type ._llcrypt registered [ 1123.634269] Key type .llcrypt registered [ 1123.859814] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1123.882721] alg: No test for adler32 (adler32-zlib) [ 1124.848647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1124.910377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1124.945722] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1125.848779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1125.861809] LNet: 49829:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1125.867494] LNetError: 49829:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1125.878618] LNet: Removed LNI 192.168.202.10@tcp [ 1125.889966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1125.931279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 1125.950901] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1133.588309] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 05:40:41 (1789378841) [ 1133.717741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1133.752008] LNet: 50429:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1133.765826] LNetError: 50429:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1133.791567] LNet: Removed LNI 192.168.202.10@tcp [ 1133.843698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1134.102240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 1141.232472] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 05:40:49 (1789378849) [ 1141.489793] LNet: 51027:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1142.089589] Key type .llcrypt unregistered [ 1142.091474] Key type ._llcrypt unregistered [ 1150.606559] Key type ._llcrypt registered [ 1150.611823] Key type .llcrypt registered [ 1150.889715] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1150.914754] alg: No test for adler32 (adler32-zlib) [ 1151.867233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1152.031714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1152.105614] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1154.020557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1154.061588] LNet: 51424:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1154.070404] LNetError: 51424:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1154.083703] LNet: Removed LNI 192.168.202.10@tcp [ 1154.159944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1154.242865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 1154.260373] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1164.803441] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 05:41:12 (1789378872) [ 1164.964489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1165.027461] LNet: 52020:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1165.044388] LNetError: 52020:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1165.093166] LNet: Removed LNI 192.168.202.10@tcp [ 1165.178342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1165.382841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 1172.651525] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 05:41:20 (1789378880) [ 1174.273555] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 1175.707657] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 05:41:24 (1789378884) [ 1177.701372] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 1180.745497] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 05:41:27 (1789378887) [ 1183.455441] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 1186.218200] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 05:41:33 (1789378893) [ 1189.056672] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1191.688620] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 05:41:38 (1789378898) [ 1193.886359] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1196.119479] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 05:41:44 (1789378904) [ 1196.430442] LNet: 54107:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1197.077830] Key type .llcrypt unregistered [ 1197.081264] Key type ._llcrypt unregistered [ 1203.973063] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1204.246601] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1204.273221] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1204.290439] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1207.164385] Key type ._llcrypt registered [ 1207.168404] Key type .llcrypt registered [ 1207.462487] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1207.475994] alg: No test for adler32 (adler32-zlib) [ 1208.522397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1208.653708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1208.673873] LNet: Added LNI 192.168.202.10@tcp [8/512/0/180] [ 1208.681343] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1208.683329] LNet: 54428:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1208.863949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1208.911805] LNet: 54447:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1208.923955] LNetError: 54447:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1208.942150] LNet: Removed LNI 10.1.2.3@tcp [ 1208.947916] LNetError: 54447:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1208.968039] LNet: Removed LNI 192.168.202.10@tcp [ 1208.990876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1209.116829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1209.201225] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1209.346951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1209.382385] LNet: 54472:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1209.391591] LNetError: 54472:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1209.411660] LNet: Removed LNI 192.168.202.10@tcp [ 1209.431375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1209.552659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1209.605122] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1209.611428] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1209.615380] LNet: 54480:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1209.760743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1209.792669] LNet: 54498:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1209.805694] LNetError: 54498:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1209.811754] LNet: Removed LNI 10.1.2.3@tcp [ 1209.814150] LNetError: 54498:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1209.835743] LNet: Removed LNI 192.168.202.10@tcp [ 1209.861291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1209.926488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1209.948436] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1209.952661] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1209.955781] LNet: 54506:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1210.146309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1210.192738] LNet: 54532:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1210.205279] LNetError: 54532:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1210.210444] LNet: Removed LNI 10.1.2.3@tcp [ 1210.220690] LNetError: 54532:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1210.276453] LNet: Removed LNI 192.168.202.10@tcp [ 1210.305171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1210.449382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1210.478905] LNet: Added LNI 192.168.202.10@tcp [8/256/0/182] [ 1210.486428] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1210.590652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1210.608807] LNet: 54560:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1210.612929] LNetError: 54560:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1210.618901] LNet: Removed LNI 10.1.2.3@tcp [ 1210.621552] LNetError: 54560:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1210.642808] LNet: Removed LNI 192.168.202.10@tcp [ 1210.659938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1210.721282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1210.755679] LNet: Added LNI 192.168.202.10@tcp [8/256/0/182] [ 1210.760591] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1210.763725] LNet: 54568:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1210.939461] LNet: Removed LNI 10.1.2.3@tcp [ 1210.949697] LNet: Removed LNI 192.168.202.10@tcp [ 1211.735726] Key type .llcrypt unregistered [ 1211.736946] Key type ._llcrypt unregistered [ 1229.134256] Key type ._llcrypt registered [ 1229.136454] Key type .llcrypt registered [ 1231.523912] Lustre: DEBUG MARKER: == sanity-lnet test 171: import --old-api with entries missing a key (LU-20595) ========================================================== 05:42:18 (1789378938) [ 1232.107410] Key type .llcrypt unregistered [ 1232.109177] Key type ._llcrypt unregistered [ 1240.621550] Key type ._llcrypt registered [ 1240.624367] Key type .llcrypt registered [ 1240.960757] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1241.018523] alg: No test for adler32 (adler32-zlib) [ 1242.029261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1242.162116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1242.198493] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1242.341636] LNet: 55641:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1242.363353] LNetError: 55641:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1242.392606] LNet: Removed LNI 192.168.202.10@tcp [ 1243.253354] Key type .llcrypt unregistered [ 1243.256940] Key type ._llcrypt unregistered [ 1256.356303] Key type ._llcrypt registered [ 1256.358652] Key type .llcrypt registered [ 1257.769281] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 05:42:45 (1789378965) [ 1259.414836] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 1261.340631] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 05:42:49 (1789378969) [ 1261.756409] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1261.773962] alg: No test for adler32 (adler32-zlib) [ 1262.722669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1262.775161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1262.796673] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1263.586962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.202.10 [ 1263.609388] LNet: 56900:0:(config.c:1740:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1263.617311] LNet: Removed LNI 192.168.202.10@tcp [ 1270.661090] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 05:42:58 (1789378978) [ 1270.972479] LNet: 57478:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1270.983649] LNetError: 57478:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1271.007495] LNet: Removed LNI 192.168.202.10@tcp [ 1271.746785] Key type .llcrypt unregistered [ 1271.750274] Key type ._llcrypt unregistered [ 1278.485307] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1278.704483] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1281.624588] Key type ._llcrypt registered [ 1281.626714] Key type .llcrypt registered [ 1281.870964] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1281.886935] alg: No test for adler32 (adler32-zlib) [ 1283.335040] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1290.431904] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 05:43:18 (1789378998) [ 1290.609413] LNet: 58395:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1290.614377] LNetError: 58395:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1290.630614] LNet: Removed LNI 10.1.2.3@tcp [ 1291.288728] Key type .llcrypt unregistered [ 1291.290301] Key type ._llcrypt unregistered [ 1299.959823] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1300.374385] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1300.386598] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1300.404093] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1303.697064] Key type ._llcrypt registered [ 1303.698384] Key type .llcrypt registered [ 1303.810662] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1303.828560] alg: No test for adler32 (adler32-zlib) [ 1304.784699] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1313.797232] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 05:43:40 (1789379020) [ 1315.769473] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1318.172930] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 05:43:46 (1789379026) [ 1318.706414] LNet: 59593:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1318.718310] LNetError: 59593:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1318.754425] LNet: Removed LNI 10.1.2.3@tcp [ 1319.638538] Key type .llcrypt unregistered [ 1319.642183] Key type ._llcrypt unregistered [ 1328.615655] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1328.929693] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1328.935936] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1328.949783] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1331.970911] Key type ._llcrypt registered [ 1331.973917] Key type .llcrypt registered [ 1332.165858] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1332.179603] alg: No test for adler32 (adler32-zlib) [ 1333.137521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1333.209725] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1344.654551] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 05:44:12 (1789379052) [ 1345.317814] LNet: 60840:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1345.324763] LNetError: 60840:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1345.339963] LNet: Removed LNI 10.1.2.3@tcp [ 1345.916302] Key type .llcrypt unregistered [ 1345.925028] Key type ._llcrypt unregistered [ 1352.090640] Key type ._llcrypt registered [ 1352.098627] Key type .llcrypt registered [ 1352.418191] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1352.432649] alg: No test for adler32 (adler32-zlib) [ 1353.402640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1353.429982] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1361.503885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.110@tcp [ 1362.476677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e local_interrupt -r 1 [ 1362.559644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e local_interrupt -r 1 [ 1362.603701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.110@tcp [ 1362.621528] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1362.655575] LNet: 1 local NIs in recovery (showing 1): 192.168.202.10@tcp [ 1362.804411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1362.839802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1362.868864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1362.920613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1362.943775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1362.965527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1362.986475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1363.015776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1363.045506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1363.063350] LNet: 61278:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1363.081399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1363.113785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1363.150679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1363.179439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1363.209773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1363.243980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1363.272156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1363.823458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e local_dropped -r 1 [ 1363.865389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e local_dropped -r 1 [ 1363.895184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.110@tcp [ 1363.931464] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1364.241383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1364.267155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1364.295706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1364.318715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1364.350709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1364.379815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1364.410703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1364.445719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1364.491314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1364.516654] LNet: 61390:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1364.530571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1364.553428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1364.577737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1364.603649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1364.638033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1364.690589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1364.749382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1365.460783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e local_aborted -r 1 [ 1365.499800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e local_aborted -r 1 [ 1365.535542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.110@tcp [ 1365.560310] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1365.570449] LNet: Skipped 1 previous similar message [ 1365.885954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1365.924486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1365.957160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1365.982658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1366.022923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1366.055573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1366.090798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1366.129912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1366.162483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1366.189914] LNet: 61501:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1366.213387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1366.265660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1366.308978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1366.351541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1366.410521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1366.456819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1366.499374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1367.131296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e local_no_route -r 1 [ 1367.176921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e local_no_route -r 1 [ 1367.235789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.110@tcp [ 1367.508971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1367.553781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1367.615519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1367.668871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1367.738178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1367.813345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1367.867259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1367.916413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1367.964839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1368.007512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1368.059588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1368.133937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1368.169847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1368.213626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1368.253748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1368.283616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1368.840496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e local_timeout -r 1 [ 1368.900671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e local_timeout -r 1 [ 1368.986670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.110@tcp [ 1369.022899] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1369.037631] LNet: Skipped 1 previous similar message [ 1369.347472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1369.376932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1369.418401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1369.447629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1369.475223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1369.510308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1369.531095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1369.570703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1369.595514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1369.613494] LNet: 61723:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1369.618404] LNet: 61723:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1369.633522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1369.665991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1369.728042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1369.760447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1369.808445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1369.887578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1369.947414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1370.631169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e local_error -r 1 [ 1370.661742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e local_error -r 1 [ 1370.693406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.110@tcp [ 1370.943523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1370.974328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1371.002837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1371.031598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1371.055394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1371.077580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1371.110435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1371.138617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1371.164905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1371.188439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1371.216414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1371.257527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1371.303569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1371.342494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1371.367731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1371.393657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1371.631649] LNet: 61876:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1371.636751] LNetError: 61876:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1371.657847] LNet: Removed LNI 192.168.202.10@tcp [ 1372.000708] Key type .llcrypt unregistered [ 1372.002910] Key type ._llcrypt unregistered [ 1390.180854] Key type ._llcrypt registered [ 1390.183690] Key type .llcrypt registered [ 1391.619787] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 05:44:59 (1789379099) [ 1392.705594] Key type .llcrypt unregistered [ 1392.711164] Key type ._llcrypt unregistered [ 1399.420410] Key type ._llcrypt registered [ 1399.422185] Key type .llcrypt registered [ 1399.686523] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1399.724634] alg: No test for adler32 (adler32-zlib) [ 1400.707567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1400.764354] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1408.369691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.110@tcp [ 1420.863232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1420.895503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1420.910216] LNet: Added LNI 192.168.202.10@tcp1 [8/256/0/180] [ 1422.295780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e local_interrupt -r 1 [ 1422.374238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp1 -m GET -e local_interrupt -r 1 [ 1422.419465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e local_interrupt -r 1 [ 1422.491407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp1 -m GET -e local_interrupt -r 1 [ 1422.611166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp -m GET -e local_interrupt -r 1 [ 1422.744450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp1 -m GET -e local_interrupt -r 1 [ 1422.805631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp -m GET -e local_interrupt -r 1 [ 1422.884807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp1 -m GET -e local_interrupt -r 1 [ 1422.983472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.110@tcp [ 1423.033025] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1423.054153] LNet: 1 local NIs in recovery (showing 1): 192.168.202.10@tcp [ 1423.379876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1423.430802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1423.478862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1423.522557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1423.577201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1423.649902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1423.687630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1423.735694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1423.820662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1423.859988] LNet: 63354:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1423.886190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1423.939103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1423.993849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1424.061936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1424.135344] LNet: There was an unexpected network error while writing to 192.168.202.10: rc = -22 [ 1424.151289] LNet: Skipped 3 previous similar messages [ 1424.164381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1424.249761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1424.330920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1425.074574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e local_dropped -r 1 [ 1425.145903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp1 -m GET -e local_dropped -r 1 [ 1425.240950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e local_dropped -r 1 [ 1425.296660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp1 -m GET -e local_dropped -r 1 [ 1425.370565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp -m GET -e local_dropped -r 1 [ 1425.456558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp1 -m GET -e local_dropped -r 1 [ 1425.533918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp -m GET -e local_dropped -r 1 [ 1425.611829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp1 -m GET -e local_dropped -r 1 [ 1425.679177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.110@tcp [ 1425.702530] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1426.029150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1426.074805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1426.119701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1426.175585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1426.228928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1426.283244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1426.310635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1426.348102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1426.398746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1426.415733] LNet: 63478:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1426.428515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1426.452465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1426.482709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1426.503637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1426.549176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1426.590444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1426.616152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1427.184583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e local_aborted -r 1 [ 1427.219826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp1 -m GET -e local_aborted -r 1 [ 1427.248655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e local_aborted -r 1 [ 1427.272160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp1 -m GET -e local_aborted -r 1 [ 1427.307509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp -m GET -e local_aborted -r 1 [ 1427.345705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp1 -m GET -e local_aborted -r 1 [ 1427.379945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp -m GET -e local_aborted -r 1 [ 1427.423599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp1 -m GET -e local_aborted -r 1 [ 1427.462983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.110@tcp [ 1427.861667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1427.898698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1427.928705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1427.957534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1428.003675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1428.033907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1428.065205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1428.112343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1428.140181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1428.188656] LNet: 63601:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1428.219151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1428.306360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1428.364899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1428.423851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1428.475988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1428.555183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1428.634599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1429.167239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e local_no_route -r 1 [ 1429.227808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp1 -m GET -e local_no_route -r 1 [ 1429.277598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e local_no_route -r 1 [ 1429.371800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp1 -m GET -e local_no_route -r 1 [ 1429.438318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp -m GET -e local_no_route -r 1 [ 1429.482720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp1 -m GET -e local_no_route -r 1 [ 1429.547762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp -m GET -e local_no_route -r 1 [ 1429.624525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp1 -m GET -e local_no_route -r 1 [ 1429.708654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.110@tcp [ 1429.747311] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1429.757673] LNet: Skipped 5 previous similar messages [ 1430.160911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1430.204422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1430.235665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1430.274377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1430.307038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1430.347509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1430.409393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1430.468773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1430.529381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1430.562201] LNet: 63725:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1430.621872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1430.708855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1430.769908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1430.870878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1430.938705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1430.966691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1431.001459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1431.431919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e local_timeout -r 1 [ 1431.472928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp1 -m GET -e local_timeout -r 1 [ 1431.510466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e local_timeout -r 1 [ 1431.544131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp1 -m GET -e local_timeout -r 1 [ 1431.591073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp -m GET -e local_timeout -r 1 [ 1431.615752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp1 -m GET -e local_timeout -r 1 [ 1431.638551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp -m GET -e local_timeout -r 1 [ 1431.662374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp1 -m GET -e local_timeout -r 1 [ 1431.692592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.110@tcp [ 1431.970939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1432.007633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1432.088411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1432.168343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1432.208890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1432.232571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1432.252137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1432.266898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1432.284692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1432.301944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1432.325367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1432.344335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1432.362535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1432.381371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1432.402244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1432.422337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1432.938279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e local_error -r 1 [ 1433.007991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp1 -m GET -e local_error -r 1 [ 1433.108977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e local_error -r 1 [ 1433.158755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp1 -m GET -e local_error -r 1 [ 1433.203395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp -m GET -e local_error -r 1 [ 1433.245463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp1 -m GET -e local_error -r 1 [ 1433.271438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp -m GET -e local_error -r 1 [ 1433.340206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp1 -m GET -e local_error -r 1 [ 1433.393331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.110@tcp [ 1433.775254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1433.809427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1433.833593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1433.862776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1433.902568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1433.989906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1434.077468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1434.158266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1434.258668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1434.337991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1434.415850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1434.498322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1434.585337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1434.640881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1434.679142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1434.729477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1440.583936] LNet: 64047:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1440.590213] LNetError: 64047:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1440.597927] LNet: Removed LNI 192.168.202.10@tcp [ 1441.649389] LNet: Removed LNI 192.168.202.10@tcp1 [ 1442.220258] Key type .llcrypt unregistered [ 1442.230525] Key type ._llcrypt unregistered [ 1460.719386] Key type ._llcrypt registered [ 1460.721277] Key type .llcrypt registered [ 1462.323561] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 05:46:10 (1789379170) [ 1463.610334] Key type .llcrypt unregistered [ 1463.611983] Key type ._llcrypt unregistered [ 1473.875058] Key type ._llcrypt registered [ 1473.876670] Key type .llcrypt registered [ 1474.045826] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1474.054469] alg: No test for adler32 (adler32-zlib) [ 1475.003724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1475.030324] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1484.066087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.110@tcp [ 1484.894353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e remote_dropped -r 1 [ 1484.930515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e remote_dropped -r 1 [ 1484.963299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.110@tcp [ 1484.981629] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1485.170121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1485.215534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1485.234310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1485.256366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1485.276990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1485.301584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1485.335810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1485.400180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1485.434930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1485.450124] LNet: 65309:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1485.468248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1485.491349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1485.507435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1485.528511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1485.562923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1485.593743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1485.621563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1486.052201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e remote_error -r 1 [ 1486.072445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e remote_error -r 1 [ 1486.107814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.110@tcp [ 1486.142574] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1486.290814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1486.312037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1486.337977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1486.360806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1486.408774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1486.443863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1486.471615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1486.494096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1486.517595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1486.531891] LNet: 65420:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1486.579092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1486.632772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1486.649622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1486.673879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1486.715285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1486.754503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1486.801033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1487.574378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e remote_timeout -r 1 [ 1487.599553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e remote_timeout -r 1 [ 1487.623670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.110@tcp [ 1487.636541] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1487.837246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1487.870521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1487.903163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1487.926095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1487.959995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1487.993355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1488.030270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1488.063723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1488.089872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1488.108651] LNet: 65532:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1488.125211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1488.168182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1488.195676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1488.228811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1488.268868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1488.297358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1488.380594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1488.800459] LNet: 65573:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1488.823581] LNetError: 65573:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1489.897809] LNet: Removed LNI 192.168.202.10@tcp [ 1490.517609] Key type .llcrypt unregistered [ 1490.522662] Key type ._llcrypt unregistered [ 1507.762491] Key type ._llcrypt registered [ 1507.765359] Key type .llcrypt registered [ 1509.146413] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 05:46:57 (1789379217) [ 1510.209444] Key type .llcrypt unregistered [ 1510.212358] Key type ._llcrypt unregistered [ 1517.348515] Key type ._llcrypt registered [ 1517.352778] Key type .llcrypt registered [ 1517.572601] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1517.585705] alg: No test for adler32 (adler32-zlib) [ 1518.549966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1518.585387] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1526.382694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.110@tcp [ 1534.899430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1534.965086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1535.001477] LNet: Added LNI 192.168.202.10@tcp1 [8/256/0/180] [ 1536.675401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e remote_dropped -r 1 [ 1536.702640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp1 -m GET -e remote_dropped -r 1 [ 1536.729437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e remote_dropped -r 1 [ 1536.756666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp1 -m GET -e remote_dropped -r 1 [ 1536.786406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp -m GET -e remote_dropped -r 1 [ 1536.826886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp1 -m GET -e remote_dropped -r 1 [ 1536.879573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp -m GET -e remote_dropped -r 1 [ 1536.914438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp1 -m GET -e remote_dropped -r 1 [ 1536.958815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.110@tcp [ 1536.985585] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1536.996137] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.110@tcp [ 1537.215839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1537.258581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1537.311324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1537.344753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1537.380742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1537.437348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1537.483588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1537.541917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1537.582426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1537.603825] LNet: 67051:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1537.622976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1537.654251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1537.678868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1537.707398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1537.736654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1537.761821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1537.785910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1537.830567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1538.324264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e remote_error -r 1 [ 1538.375386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp1 -m GET -e remote_error -r 1 [ 1538.451362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e remote_error -r 1 [ 1538.485310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp1 -m GET -e remote_error -r 1 [ 1538.515649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp -m GET -e remote_error -r 1 [ 1538.551334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp1 -m GET -e remote_error -r 1 [ 1538.603223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp -m GET -e remote_error -r 1 [ 1538.670224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp1 -m GET -e remote_error -r 1 [ 1538.709586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.110@tcp [ 1538.729576] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1538.738459] LNet: Skipped 3 previous similar messages [ 1539.003151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1539.042591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1539.089028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1539.124342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1539.194281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1539.258725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1539.319429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1539.402451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1539.451698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1539.486673] LNet: 67176:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1539.511261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1539.552954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1539.583916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1539.609720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1539.638652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1539.681884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1539.738302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1539.811251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1540.368299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e remote_timeout -r 1 [ 1540.402848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp1 -m GET -e remote_timeout -r 1 [ 1540.443098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e remote_timeout -r 1 [ 1540.483969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp1 -m GET -e remote_timeout -r 1 [ 1540.559182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp -m GET -e remote_timeout -r 1 [ 1540.604377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp1 -m GET -e remote_timeout -r 1 [ 1540.640735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp -m GET -e remote_timeout -r 1 [ 1540.696187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp1 -m GET -e remote_timeout -r 1 [ 1540.763396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.110@tcp [ 1540.780764] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1540.789329] LNet: Skipped 1 previous similar message [ 1541.046624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1541.109337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1541.143362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1541.165338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1541.197292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1541.225204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1541.259254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1541.289385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1541.318856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1541.349534] LNet: 67301:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1541.382075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1541.427894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1541.462341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1541.488329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1541.537738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1541.574343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1541.619880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1541.656703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1546.554422] LNet: 67379:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1546.560809] LNetError: 67379:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1546.569828] LNet: Removed LNI 192.168.202.10@tcp [ 1546.588902] LNet: Removed LNI 192.168.202.10@tcp1 [ 1546.995534] Key type .llcrypt unregistered [ 1546.999588] Key type ._llcrypt unregistered [ 1563.662438] Key type ._llcrypt registered [ 1563.667105] Key type .llcrypt registered [ 1565.030545] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 05:47:53 (1789379273) [ 1567.001786] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1568.806757] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 05:47:56 (1789379276) [ 1570.009594] Key type .llcrypt unregistered [ 1570.012946] Key type ._llcrypt unregistered [ 1576.825274] Key type ._llcrypt registered [ 1576.826753] Key type .llcrypt registered [ 1577.005896] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1577.025302] alg: No test for adler32 (adler32-zlib) [ 1578.246835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1578.316875] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1587.937430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.110@tcp [ 1588.762584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e network_timeout -r 1 [ 1588.816966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e network_timeout -r 1 [ 1588.854565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.110@tcp [ 1588.871442] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1589.139887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1589.189805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1589.247232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1589.299864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1589.363694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1589.400867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1589.450594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1589.477857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1589.528600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1589.544391] LNet: 68953:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1589.564475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1589.601983] LNet: 1 local NIs in recovery (showing 1): 192.168.202.10@tcp [ 1589.607913] LNet: There was an unexpected network error while writing to 192.168.202.10: rc = -22 [ 1589.619619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1589.676630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1589.705859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1589.740731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1589.768782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1590.035603] LNet: 68992:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1590.050786] LNetError: 68992:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1590.064909] LNet: Removed LNI 192.168.202.10@tcp [ 1590.424520] Key type .llcrypt unregistered [ 1590.428360] Key type ._llcrypt unregistered [ 1610.035330] Key type ._llcrypt registered [ 1610.041429] Key type .llcrypt registered [ 1610.284975] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1610.345808] alg: No test for adler32 (adler32-zlib) [ 1611.270913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1611.298077] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1618.682162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.110@tcp [ 1627.538194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1627.606973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1627.627615] LNet: Added LNI 192.168.202.10@tcp1 [8/256/0/180] [ 1628.943276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e network_timeout -r 1 [ 1628.966990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp1 -m GET -e network_timeout -r 1 [ 1628.998410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e network_timeout -r 1 [ 1629.031279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp1 -m GET -e network_timeout -r 1 [ 1629.071915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp -m GET -e network_timeout -r 1 [ 1629.137544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp1 -m GET -e network_timeout -r 1 [ 1629.180274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp -m GET -e network_timeout -r 1 [ 1629.219832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp1 -m GET -e network_timeout -r 1 [ 1629.291281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.110@tcp [ 1629.317389] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1629.587786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1629.627679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1629.672144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1629.713982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1629.762554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1629.793630] LNet: 1 local NIs in recovery (showing 1): 192.168.202.10@tcp [ 1629.795239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1629.796752] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.110@tcp [ 1629.852197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1629.885265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1629.921750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1629.950418] LNet: 69874:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1629.971469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1630.003462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1630.047512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1630.128846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1630.191557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1630.268302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1635.709735] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1636.139335] LNet: 69948:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1636.145698] LNetError: 69948:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1636.151815] LNet: Removed LNI 192.168.202.10@tcp [ 1636.176752] LNet: Removed LNI 192.168.202.10@tcp1 [ 1636.650466] Key type .llcrypt unregistered [ 1636.651958] Key type ._llcrypt unregistered [ 1661.458580] Key type ._llcrypt registered [ 1661.462517] Key type .llcrypt registered [ 1662.896714] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 05:49:30 (1789379370) [ 1663.239958] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1663.247717] alg: No test for adler32 (adler32-zlib) [ 1664.392917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1664.494763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1664.531102] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1665.807942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1665.871332] LNet: Added LNI 192.168.202.10@tcp1 [8/256/0/180] [ 1666.910497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 1667.048990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1667.108329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1667.174679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1667.214975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1667.263988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1667.314681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.10@tcp [ 1667.349992] LNetError: 71055:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.202.10@tcp to recovery [ 1667.551587] LNet: 1 local NIs in recovery (showing 1): 192.168.202.10@tcp [ 1667.556749] LNet: There was an unexpected network error while writing to 192.168.202.10: rc = -22 [ 1669.611302] LNet: There was an unexpected network error while writing to 192.168.202.10: rc = -22 [ 1669.854233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1673.700563] LNet: There was an unexpected network error while writing to 192.168.202.10: rc = -22 [ 1674.524808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1674.556609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1674.622724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1674.636935] LNet: 71162:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1674.641407] LNetError: 71162:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1674.645952] LNet: Removed LNI 192.168.202.10@tcp [ 1674.654439] LNet: Removed LNI 192.168.202.10@tcp1 [ 1674.667231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1674.699801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1674.718527] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1675.239227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1675.254588] LNet: Added LNI 192.168.202.10@tcp1 [8/256/0/180] [ 1676.023559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 1676.085367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1676.160949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1676.229789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1676.266896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1676.312372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1676.342416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1676.405942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.10@tcp [ 1676.438139] LNetError: 71435:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.202.10@tcp to recovery [ 1676.767595] LNet: There was an unexpected network error while writing to 192.168.202.10: rc = -22 [ 1680.026727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1682.911646] LNet: There was an unexpected network error while writing to 192.168.202.10: rc = -22 [ 1682.926672] LNet: Skipped 1 previous similar message [ 1688.164380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1688.197387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1688.228602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1695.960924] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 05:50:04 (1789379404) [ 1696.092665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1696.136888] LNet: 72154:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1696.149692] LNetError: 72154:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1696.161028] LNetError: 72154:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1696.172969] LNet: Removed LNI 192.168.202.10@tcp [ 1696.190811] LNet: Removed LNI 192.168.202.10@tcp1 [ 1696.280828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1696.344024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1696.380171] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1697.259368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1697.282400] LNet: Added LNI 192.168.202.10@tcp1 [8/256/0/180] [ 1698.222404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 1698.317612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1698.343952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1698.378885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1698.411886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1698.437953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1698.468727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.10@tcp --health 0 [ 1698.570319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1699.362743] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.10@tcp [ 1699.366629] LNet: There was an unexpected network error while writing to 192.168.202.10: rc = -22 [ 1699.384438] LNet: Skipped 1 previous similar message [ 1714.561843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1714.858919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1714.898759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1714.937755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.10@tcp --health 500 [ 1715.115719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1722.031202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1722.141600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1722.169718] LNet: 72654:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1722.177978] LNetError: 72654:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1722.186595] LNetError: 72654:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1722.194964] LNet: Removed LNI 192.168.202.10@tcp [ 1722.213674] LNet: Removed LNI 192.168.202.10@tcp1 [ 1722.230614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1722.271598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1722.300586] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1723.444740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1723.510908] LNet: Added LNI 192.168.202.10@tcp1 [8/256/0/180] [ 1724.543691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 1724.583477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1724.686383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1724.728596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1724.782153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1724.817425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1724.905984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1724.995643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.10@tcp --health 0 [ 1726.367616] LNet: There was an unexpected network error while writing to 192.168.202.10: rc = -22 [ 1726.370383] LNet: Skipped 3 previous similar messages [ 1727.731786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1737.237685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1737.325419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1737.380059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1745.734338] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 05:50:53 (1789379453) [ 1746.172387] LNet: 73711:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1746.180502] LNetError: 73711:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1746.186584] LNetError: 73711:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1746.191780] LNet: Removed LNI 192.168.202.10@tcp [ 1746.206369] LNet: Removed LNI 192.168.202.10@tcp1 [ 1746.820084] Key type .llcrypt unregistered [ 1746.822344] Key type ._llcrypt unregistered [ 1754.051486] Key type ._llcrypt registered [ 1754.055985] Key type .llcrypt registered [ 1754.268336] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1754.290806] alg: No test for adler32 (adler32-zlib) [ 1755.268506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1755.348986] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1765.993597] Lustre: DEBUG MARKER: Initial discovery [ 1766.266024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.110@tcp [ 1769.374205] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1769.449753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1769.546395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.110@tcp [ 1769.609984] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1769.644417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1769.699111] LNet: 1 local NIs in recovery (showing 1): 192.168.202.10@tcp [ 1769.714294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.202.110@tcp -r 1 -m GET -l 3 [ 1769.775193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.202.110@tcp -r 1 -m GET -e local_error [ 1769.840542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.202.110@tcp -r 1 -m PUT -l 6 [ 1771.408510] Lustre: DEBUG MARKER: Force oleg210-client.virtnet to discover 192.168.202.110@tcp (in background) [ 1771.538642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.110@tcp [ 1773.347992] Lustre: DEBUG MARKER: Force oleg210-server to discover 192.168.202.10@tcp [ 1774.630191] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1777.070748] Lustre: DEBUG MARKER: Wait for 74476 [ 1777.759179] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1778.411390] Lustre: DEBUG MARKER: Finished wait on 74476 [ 1778.673750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1778.720540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1779.078488] LNet: 74941:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1779.087588] LNetError: 74941:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1779.112384] LNet: Removed LNI 192.168.202.10@tcp [ 1779.615518] Key type .llcrypt unregistered [ 1779.617486] Key type ._llcrypt unregistered [ 1798.492675] Key type ._llcrypt registered [ 1798.497027] Key type .llcrypt registered [ 1800.084568] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 05:51:47 (1789379507) [ 1800.661131] Key type .llcrypt unregistered [ 1800.664165] Key type ._llcrypt unregistered [ 1806.810474] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1806.982718] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1806.989852] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1806.998654] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1809.594191] Key type ._llcrypt registered [ 1809.596402] Key type .llcrypt registered [ 1809.842523] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1809.865175] alg: No test for adler32 (adler32-zlib) [ 1810.919564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1811.045854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1811.083036] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1812.087761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1812.165289] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1822.252520] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 05:52:10 (1789379530) [ 1822.394734] LNet: 77030:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1822.400941] LNetError: 77030:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1822.408397] LNet: Removed LNI 10.1.2.3@tcp [ 1822.426767] LNet: Removed LNI 192.168.202.10@tcp [ 1823.220902] Key type .llcrypt unregistered [ 1823.226968] Key type ._llcrypt unregistered [ 1831.868579] Key type ._llcrypt registered [ 1831.871717] Key type .llcrypt registered [ 1832.243649] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1832.280664] alg: No test for adler32 (adler32-zlib) [ 1833.495353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1833.672623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1833.727924] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1835.154211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1835.206475] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1836.088413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.202.10@tcp [ 1837.151248] LNetError: Unexpected error -22 connecting to 192.168.202.10@tcp at host 192.168.202.10:988 [ 1845.018202] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 05:52:32 (1789379552) [ 1845.242320] LNet: 78170:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1845.256294] LNetError: 78170:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1845.280558] LNet: Removed LNI 10.1.2.3@tcp [ 1845.306453] LNet: Removed LNI 192.168.202.10@tcp [ 1846.203596] Key type .llcrypt unregistered [ 1846.206795] Key type ._llcrypt unregistered [ 1855.124405] Key type ._llcrypt registered [ 1855.126300] Key type .llcrypt registered [ 1855.394429] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1855.428970] alg: No test for adler32 (adler32-zlib) [ 1856.455696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1856.513363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1856.553430] LNet: Added LNI 192.168.202.10@tcp1 [8/256/0/180] [ 1857.414196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1857.429751] LNet: Added LNI 192.168.202.10@tcp2 [8/256/0/180] [ 1858.942878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.202.10@tcp1 --nid 192.168.202.10@tcp2 [ 1865.137755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.10@tcp1 192.168.202.10@tcp2 [ 1865.169668] LNetError: 79012:0:(lib-move.c:2094:lnet_handle_find_routed_path()) Unable to send message from 192.168.202.10@tcp1 to 192.168.202.10@tcp2 - Route table may be misconfigured [ 1865.205949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.10@tcp2 192.168.202.10@tcp1 [ 1874.653452] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 05:53:01 (1789379581) [ 1874.865514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1874.927691] LNet: 79594:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1874.951782] LNetError: 79594:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1874.962302] LNet: Removed LNI 192.168.202.10@tcp1 [ 1876.015215] LNet: Removed LNI 192.168.202.10@tcp2 [ 1876.083699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1876.245546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1876.297274] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1877.488540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1877.517121] LNet: Added LNI 192.168.202.10@tcp1 [8/256/0/180] [ 1879.076473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 1879.135713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.10@tcp 192.168.202.10@tcp [ 1879.180896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.10@tcp1 192.168.202.10@tcp1 [ 1879.219901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.10@tcp -d 192.168.202.10@tcp -e network_timeout [ 1879.256527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.10@tcp -d 192.168.202.10@tcp1 -e network_timeout [ 1879.298921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp -e network_timeout [ 1879.334469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp1 -e network_timeout [ 1879.370702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 1879.389258] LNet: There was an unexpected network error while writing to 192.168.202.10: rc = -22 [ 1879.409566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1879.546479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1879.713236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1880.287349] LNet: 1 local NIs in recovery (showing 1): 192.168.202.10@tcp [ 1889.173526] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 05:53:16 (1789379596) [ 1889.301134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1889.340689] LNet: 80461:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1889.355890] LNetError: 80461:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1889.359298] LNetError: 80461:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1889.374803] LNet: Removed LNI 192.168.202.10@tcp [ 1890.414414] LNet: Removed LNI 192.168.202.10@tcp1 [ 1890.525047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1890.686985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1891.320717] LNet: 80502:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1892.295394] Key type .llcrypt unregistered [ 1892.299119] Key type ._llcrypt unregistered [ 1911.172024] Key type ._llcrypt registered [ 1911.176376] Key type .llcrypt registered [ 1912.886756] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 05:53:40 (1789379620) [ 1913.295664] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1913.331415] alg: No test for adler32 (adler32-zlib) [ 1914.369488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1915.652629] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1917.708327] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 05:53:45 (1789379625) [ 1917.797480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1917.816267] LNet: 81608:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1917.839749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1917.908597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1917.943922] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1918.650177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1918.658254] LNet: Added LNI 192.168.202.10@tcp1 [8/256/0/180] [ 1919.210270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 1919.236870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp1 [ 1919.259868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp1 [ 1926.596776] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 05:53:54 (1789379634) [ 1928.722489] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg210-server [ 1930.307059] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 05:53:58 (1789379638) [ 1932.536824] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg210-server [ 1934.336412] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 05:54:02 (1789379642) [ 1936.445992] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg210-server [ 1938.297796] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 05:54:06 (1789379646) [ 1939.961620] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg210-server [ 1941.575505] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 05:54:09 (1789379649) [ 1944.944251] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg210-server [ 1947.051496] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 05:54:14 (1789379654) [ 1950.281433] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg210-server [ 1952.862821] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 05:54:20 (1789379660) [ 1955.719908] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg210-server [ 1958.091671] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 05:54:25 (1789379665) [ 1960.982872] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg210-server [ 1963.950092] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 05:54:31 (1789379671) [ 1966.376630] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg210-server [ 1968.846715] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 05:54:36 (1789379676) [ 1968.961554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1968.987867] LNet: 85549:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1968.995154] LNetError: 85549:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1969.002946] LNet: Removed LNI 192.168.202.10@tcp [ 1969.018266] LNet: Removed LNI 192.168.202.10@tcp1 [ 1969.071809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1969.158671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1969.182152] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1970.101392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1970.183351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 1970.329455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1970.352199] LNet: 85703:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1970.355638] LNetError: 85703:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1970.360139] LNetError: 85703:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1970.372881] LNet: Removed LNI 192.168.202.10@tcp [ 1970.394671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1970.461666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1970.499706] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1971.859979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 1972.202788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1972.262282] LNetError: 85848:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1972.296713] LNet: Removed LNI 192.168.202.10@tcp [ 1972.337897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1972.398313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1972.434326] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1973.438030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1973.579656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 1973.863836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1973.907275] LNet: 85996:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1973.920225] LNet: 85996:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1974.956818] LNet: Removed LNI 192.168.202.10@tcp [ 1975.011775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1975.117154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1975.171387] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1976.856686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 1977.198122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1977.235306] LNetError: 86141:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1977.241360] LNetError: 86141:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1977.266254] LNet: Removed LNI 192.168.202.10@tcp [ 1977.282571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1977.432169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1977.483980] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1977.987719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1978.054962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 1978.274541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1978.303738] LNet: 86288:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1978.310619] LNet: 86288:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1979.364925] LNet: Removed LNI 192.168.202.10@tcp [ 1979.387947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1979.479305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1979.497051] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1980.110262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 1980.252580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1981.291246] LNet: Removed LNI 192.168.202.10@tcp [ 1981.351993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1981.447418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1981.467439] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1982.209832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1982.292570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 1982.448393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1982.463995] LNetError: 86581:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1982.469140] LNetError: 86581:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1982.488041] LNet: Removed LNI 192.168.202.10@tcp [ 1982.499117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1982.530459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1982.561498] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1983.239483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 1984.575180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1985.638252] LNet: Removed LNI 192.168.202.10@tcp [ 1985.657680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1985.706239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1985.747937] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1986.661288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1986.768076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 1987.072978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1987.117071] LNet: 86883:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1987.127109] LNet: 86883:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1988.214741] LNet: Removed LNI 192.168.202.10@tcp [ 1988.260563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1988.294548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1988.311353] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1989.282525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 1990.754851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1990.775532] LNetError: 87038:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1990.779471] LNetError: 87038:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1991.846947] LNet: Removed LNI 192.168.202.10@tcp [ 1991.946926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1992.065595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1992.095934] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1993.867140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1994.131530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 1994.815412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1995.957514] LNet: Removed LNI 192.168.202.10@tcp [ 1996.073251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1996.182470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1996.266484] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1997.896951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 1998.391977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1999.461890] LNet: Removed LNI 192.168.202.10@tcp [ 1999.557546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1999.775713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1999.829542] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2000.789504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 2000.866536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2000.926695] LNet: Removed LNI 192.168.202.10@tcp [ 2000.953671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2001.023593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2001.081793] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2002.161297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 2002.243330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2002.268732] LNet: Removed LNI 192.168.202.10@tcp [ 2002.305800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2002.399413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 2002.537171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2002.590710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2002.649266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 2002.726349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2002.761482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2002.814650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 2002.883668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2002.921528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2003.007848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 2003.046362] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2004.118933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2004.136895] LNet: 87771:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2004.141304] LNet: 87771:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 2004.151826] LNet: Removed LNI 192.168.202.10@tcp [ 2004.165618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2004.249131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2012.201736] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 05:55:19 (1789379719) [ 2012.326282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2012.393987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2012.513052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 2012.572725] LNet: Added LNI 192.168.202.10@tcp231 [8/256/0/180] [ 2014.573495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2014.637684] LNetError: 88499:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2014.650529] LNetError: 88499:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2014.696204] LNet: Removed LNI 192.168.202.10@tcp231 [ 2014.741370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2014.845549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 2014.901213] LNet: Added LNI 192.168.202.10@tcp231 [8/256/0/180] [ 2014.912787] LNet: 88508:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2015.711472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 2015.775972] LNet: Removed LNI 192.168.202.10@tcp231 [ 2015.819915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 2015.888417] LNet: Added LNI 192.168.202.10@tcp231 [8/256/0/180] [ 2016.706418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2016.728254] LNet: Removed LNI 192.168.202.10@tcp231 [ 2016.743842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2016.783121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 2016.799768] LNet: Added LNI 192.168.202.10@tcp231 [8/256/0/0] [ 2016.804693] LNet: 88660:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2017.052760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 2017.074649] LNet: Removed LNI 192.168.202.10@tcp231 [ 2017.088034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 2017.108368] LNet: Added LNI 192.168.202.10@tcp231 [8/256/0/0] [ 2017.835926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2017.858407] LNet: Removed LNI 192.168.202.10@tcp231 [ 2017.873565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2017.933617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 2017.966691] LNet: Added LNI 192.168.202.10@tcp231 [8/256/0/60] [ 2017.975261] LNet: 88811:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2018.348988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 2018.404442] LNet: Removed LNI 192.168.202.10@tcp231 [ 2018.452840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 2018.523274] LNet: Added LNI 192.168.202.10@tcp231 [8/256/0/60] [ 2019.998455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2020.031589] LNet: Removed LNI 192.168.202.10@tcp231 [ 2020.074572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2020.170846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 2020.202975] LNet: Added LNI 192.168.202.10@tcp231 [8/256/0/180] [ 2020.208883] LNet: 88963:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2020.762935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 2020.807438] LNet: Removed LNI 192.168.202.10@tcp231 [ 2020.841076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 2020.900804] LNet: Added LNI 192.168.202.10@tcp231 [8/256/0/180] [ 2032.197839] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 05:55:39 (1789379739) [ 2032.786963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2032.869142] LNet: Removed LNI 192.168.202.10@tcp231 [ 2032.951344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2033.023563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 2033.056540] LNet: Added LNI 192.168.202.10@tcp232 [8/256/0/180] [ 2033.780369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 2033.799948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 2033.817352] LNet: Removed LNI 192.168.202.10@tcp232 [ 2033.829271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 2033.852576] LNet: Added LNI 192.168.202.10@tcp232 [8/256/0/180] [ 2034.370203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 2034.398129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 2034.425224] LNet: Removed LNI 192.168.202.10@tcp232 [ 2034.441728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 2034.481906] LNet: Added LNI 192.168.202.10@tcp232 [8/256/0/180] [ 2035.234600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 2035.268994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 2035.294268] LNet: Removed LNI 192.168.202.10@tcp232 [ 2035.308297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 2035.335453] LNet: Added LNI 192.168.202.10@tcp232 [8/256/0/180] [ 2036.259447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 2036.313151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 2036.357633] LNet: Removed LNI 192.168.202.10@tcp232 [ 2036.399963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 2045.382491] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 05:55:52 (1789379752) [ 2045.972469] LNet: 90820:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2045.977701] LNet: 90820:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 6 previous similar messages [ 2046.877192] Key type .llcrypt unregistered [ 2046.878576] Key type ._llcrypt unregistered [ 2054.914708] Key type ._llcrypt registered [ 2054.916196] Key type .llcrypt registered [ 2055.104210] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2055.115938] alg: No test for adler32 (adler32-zlib) [ 2056.349397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2056.412805] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2067.258775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.110@tcp [ 2080.118252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2080.166667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2080.197648] LNet: Added LNI 192.168.202.10@tcp1 [8/256/0/180] [ 2083.125886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e local_interrupt -r 2 [ 2083.198775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp1 -m GET -e local_interrupt -r 2 [ 2083.254865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e local_interrupt -r 2 [ 2083.339565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp1 -m GET -e local_interrupt -r 2 [ 2083.388283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp -m GET -e local_interrupt -r 2 [ 2083.464283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp1 -m GET -e local_interrupt -r 2 [ 2083.520293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp -m GET -e local_interrupt -r 2 [ 2083.578200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp1 -m GET -e local_interrupt -r 2 [ 2083.689738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.10@tcp 192.168.202.110@tcp [ 2083.725201] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 2083.735778] LNet: 1 local NIs in recovery (showing 1): 192.168.202.10@tcp [ 2084.175945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2084.234566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2084.290692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2084.389742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2084.455079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2084.502988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2084.554992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2084.611894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2084.674151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2084.733374] LNet: 91486:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2084.785661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2084.859409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2084.919032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2085.022685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2085.135339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2085.205181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2086.094354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e local_dropped -r 2 [ 2086.164585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp1 -m GET -e local_dropped -r 2 [ 2086.251185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e local_dropped -r 2 [ 2086.363453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp1 -m GET -e local_dropped -r 2 [ 2086.476358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp -m GET -e local_dropped -r 2 [ 2086.630341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp1 -m GET -e local_dropped -r 2 [ 2086.753778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp -m GET -e local_dropped -r 2 [ 2086.868564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp1 -m GET -e local_dropped -r 2 [ 2086.990571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.10@tcp 192.168.202.110@tcp [ 2087.078714] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 2087.797102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2087.848506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2087.926306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2087.991598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2088.066360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2088.144565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2088.242151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2088.333933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2088.471196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2088.540346] LNet: 91609:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2088.589155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2088.644791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2088.761778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2088.898542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2089.023676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2089.167197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2090.173612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e local_aborted -r 2 [ 2090.232690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp1 -m GET -e local_aborted -r 2 [ 2090.314887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e local_aborted -r 2 [ 2090.454752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp1 -m GET -e local_aborted -r 2 [ 2090.581249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp -m GET -e local_aborted -r 2 [ 2090.723424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp1 -m GET -e local_aborted -r 2 [ 2090.829538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp -m GET -e local_aborted -r 2 [ 2090.925066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp1 -m GET -e local_aborted -r 2 [ 2091.031711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.10@tcp 192.168.202.110@tcp [ 2091.086277] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 2091.722591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2091.793538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2091.909431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2091.971527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2091.987798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2092.042654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2092.097481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2092.171404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2092.237056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2092.319685] LNet: 91731:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2092.441226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2092.587618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2092.754231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2092.892876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2092.955917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2093.010954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2094.107992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e local_no_route -r 2 [ 2094.178913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp1 -m GET -e local_no_route -r 2 [ 2094.286948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e local_no_route -r 2 [ 2094.417530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp1 -m GET -e local_no_route -r 2 [ 2094.517396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp -m GET -e local_no_route -r 2 [ 2094.626461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp1 -m GET -e local_no_route -r 2 [ 2094.739540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp -m GET -e local_no_route -r 2 [ 2094.836905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp1 -m GET -e local_no_route -r 2 [ 2094.940579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.10@tcp 192.168.202.110@tcp [ 2094.995852] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 2095.493357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2095.556106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2095.615620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2095.679056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2095.773834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2095.881170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2096.026763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2096.111481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2096.222695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2096.280714] LNet: 91854:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2096.331978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2096.422375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2096.490899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2096.578873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2096.648423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2096.780681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2098.095778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e local_timeout -r 2 [ 2098.211213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp1 -m GET -e local_timeout -r 2 [ 2098.338925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e local_timeout -r 2 [ 2098.481844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp1 -m GET -e local_timeout -r 2 [ 2098.625656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp -m GET -e local_timeout -r 2 [ 2098.707520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp1 -m GET -e local_timeout -r 2 [ 2098.803464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp -m GET -e local_timeout -r 2 [ 2098.876759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp1 -m GET -e local_timeout -r 2 [ 2098.980437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.10@tcp 192.168.202.110@tcp [ 2099.034699] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 2099.457488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2099.510407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2099.537448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2099.570940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2099.661152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2099.733126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2099.787737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2099.820043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2099.851500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2099.890575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2099.923635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2099.965662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2100.029042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2100.089466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2100.152047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2101.180350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e remote_dropped -r 2 [ 2101.257166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp1 -m GET -e remote_dropped -r 2 [ 2101.306617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e remote_dropped -r 2 [ 2101.338927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp1 -m GET -e remote_dropped -r 2 [ 2101.404329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp -m GET -e remote_dropped -r 2 [ 2101.493889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp1 -m GET -e remote_dropped -r 2 [ 2101.586957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp -m GET -e remote_dropped -r 2 [ 2101.655540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp1 -m GET -e remote_dropped -r 2 [ 2101.742931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.10@tcp 192.168.202.110@tcp [ 2101.768809] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.110@tcp [ 2102.116411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2102.232422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2102.304988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2102.345834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2102.391059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2102.433511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2102.506073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2102.555131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2102.600400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2102.633720] LNet: 92099:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2102.640910] LNet: 92099:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 2102.667556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2102.724575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2102.795833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2102.862793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2102.899389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2102.929909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2108.742934] LNet: 92174:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2108.748146] LNetError: 92174:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2108.758097] LNet: Removed LNI 192.168.202.10@tcp [ 2108.777879] LNet: Removed LNI 192.168.202.10@tcp1 [ 2109.222486] Key type .llcrypt unregistered [ 2109.227733] Key type ._llcrypt unregistered [ 2129.224019] Key type ._llcrypt registered [ 2129.225584] Key type .llcrypt registered [ 2130.934874] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 05:57:18 (1789379838) [ 2131.731585] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2131.756884] alg: No test for adler32 (adler32-zlib) [ 2132.839050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2132.916084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2132.981481] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2134.028882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 2134.067472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.10@tcp [ 2134.099426] LNetError: 93162:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.202.10@tcp to recovery [ 2134.234341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.202.10@tcp [ 2134.346926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2143.446306] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 05:57:30 (1789379850) [ 2143.657059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2143.710982] LNet: 93753:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2143.723126] LNetError: 93753:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2143.761122] LNet: Removed LNI 192.168.202.10@tcp [ 2143.811343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2143.977788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2144.007192] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2145.048860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2145.087809] LNet: Added LNI 192.168.202.10@tcp1 [8/256/0/180] [ 2146.684711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.202.10@tcp [ 2146.702886] LNetError: 94027:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.202.10@tcp to recovery [ 2146.975605] LNet: 1 local NIs in recovery (showing 1): 192.168.202.10@tcp [ 2146.979266] LNet: There was an unexpected network error while writing to 192.168.202.10: rc = -22 [ 2148.014737] LNet: There was an unexpected network error while writing to 192.168.202.10: rc = -22 [ 2148.060556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.202.10@tcp [ 2163.766873] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 05:57:51 (1789379871) [ 2166.339573] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg210-server [ 2168.935601] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 05:57:56 (1789379876) [ 2170.462939] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg210-server [ 2172.489500] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 05:58:00 (1789379880) [ 2174.756799] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg210-server [ 2177.695745] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 05:58:04 (1789379884) [ 2178.078255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2178.113582] LNet: 95701:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2178.135834] LNetError: 95701:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2178.164243] LNet: Removed LNI 192.168.202.10@tcp [ 2178.195348] LNet: Removed LNI 192.168.202.10@tcp1 [ 2178.253760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2178.334765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2178.361247] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2179.896199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 2180.041227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 2180.113426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2180.303481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 2180.383487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2180.464298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 2189.700920] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 05:58:17 (1789379897) [ 2189.898411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2189.948877] LNet: 96435:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2189.982700] LNetError: 96435:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2189.988113] LNetError: 96435:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2190.111836] LNet: Removed LNI 192.168.202.10@tcp [ 2190.274311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2190.483333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2190.568933] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2201.562120] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 05:58:28 (1789379908) [ 2202.012308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2202.080937] LNet: 97153:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2202.087516] LNetError: 97153:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2202.115662] LNet: Removed LNI 192.168.202.10@tcp [ 2202.152385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2202.256835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2202.309359] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2215.088580] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 05:58:41 (1789379921) [ 2217.981740] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 2220.259781] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 05:58:47 (1789379927) [ 2220.924979] LNet: 98203:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2220.931697] LNetError: 98203:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2220.949919] LNet: Removed LNI 192.168.202.10@tcp [ 2221.464793] Key type .llcrypt unregistered [ 2221.466723] Key type ._llcrypt unregistered [ 2228.970570] Key type ._llcrypt registered [ 2228.979483] Key type .llcrypt registered [ 2229.261972] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2229.290459] alg: No test for adler32 (adler32-zlib) [ 2230.425796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2230.497628] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2240.644796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.110@tcp [ 2246.754289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.202.110@tcp [ 2247.412888] LNet: 98620:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2247.419689] LNetError: 98620:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2247.441435] LNet: Removed LNI 192.168.202.10@tcp [ 2248.089738] Key type .llcrypt unregistered [ 2248.092376] Key type ._llcrypt unregistered [ 2263.620592] Key type ._llcrypt registered [ 2263.622723] Key type .llcrypt registered [ 2265.414614] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 2267.701959] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 2269.842886] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 05:59:37 (1789379977) [ 2271.619290] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 2274.211626] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 05:59:41 (1789379981) [ 2276.470720] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg210-server [ 2278.675975] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 05:59:46 (1789379986) [ 2281.439840] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg210-server [ 2283.396567] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 05:59:51 (1789379991) [ 2286.106509] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2286.152536] alg: No test for adler32 (adler32-zlib) [ 2287.437994] Lustre: Lustre: Build Version: 2.17.58_2_g65a210d [ 2287.822614] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2289.655193] Key type lgssc registered [ 2291.489909] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2318.555040] Key type lgssc unregistered [ 2319.005691] LNet: 101860:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2319.018931] LNetError: 101860:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2319.041703] LNet: Removed LNI 192.168.202.10@tcp [ 2320.214570] Key type .llcrypt unregistered [ 2320.219493] Key type ._llcrypt unregistered [ 2343.182589] Key type ._llcrypt registered [ 2343.187493] Key type .llcrypt registered [ 2344.771490] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 06:00:52 (1789380052) [ 2345.640256] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2345.659913] alg: No test for adler32 (adler32-zlib) [ 2346.714446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2346.797603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2346.858358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2346.964713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2347.043825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2355.938140] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 06:01:03 (1789380063) [ 2356.206281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2356.230562] LNet: 103304:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2356.251942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2356.350256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2356.389234] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2357.466931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2357.494928] LNet: 103449:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2357.501099] LNetError: 103449:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2357.515358] LNet: Removed LNI 192.168.202.10@tcp [ 2357.536870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2357.626619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2357.662948] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2358.292634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2358.333753] LNetError: 103477:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2358.350830] LNet: Removed LNI 192.168.202.10@tcp [ 2358.376283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2358.455843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2358.507363] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2368.664318] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 06:01:16 (1789380076) [ 2368.864773] LNet: 104069:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2368.876401] LNet: 104069:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2368.886733] LNetError: 104069:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2368.912479] LNet: Removed LNI 192.168.202.10@tcp [ 2369.904560] Key type .llcrypt unregistered [ 2369.908872] Key type ._llcrypt unregistered [ 2378.966887] Key type ._llcrypt registered [ 2378.972186] Key type .llcrypt registered [ 2379.446654] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2379.492154] alg: No test for adler32 (adler32-zlib) [ 2380.643378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2380.800584] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2380.804248] LNet: *** cfs_fail_loc=e003, val=100*** [ 2382.340708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 2382.401455] LNet: *** cfs_fail_loc=e003, val=100*** [ 2382.406464] LNet: Skipped 2 previous similar messages [ 2382.430441] Ping reply received with mapping: [ 2382.438427] found metadata reply NI[192.168.202.10@tcp] [ 2382.451479] 0a ca a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2382.469583] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2382.476671] 5a 5a 5a 5a 5a 5a 5a 5a [ 2382.759326] LNet: 104474:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2382.770571] LNetError: 104474:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2383.804799] LNet: Removed LNI 192.168.202.10@tcp [ 2384.985611] Key type .llcrypt unregistered [ 2384.988074] Key type ._llcrypt unregistered [ 2401.784145] Key type ._llcrypt registered [ 2401.786286] Key type .llcrypt registered [ 2403.366186] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 06:01:50 (1789380110) [ 2403.934544] Key type .llcrypt unregistered [ 2403.937541] Key type ._llcrypt unregistered [ 2412.100700] Key type ._llcrypt registered [ 2412.102519] Key type .llcrypt registered [ 2412.414694] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2412.461358] alg: No test for adler32 (adler32-zlib) [ 2413.718810] LNetError: 105523:0:(api-ni.c:2855:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2414.263930] Key type .llcrypt unregistered [ 2414.272654] Key type ._llcrypt unregistered [ 2422.435930] Key type ._llcrypt registered [ 2422.439970] Key type .llcrypt registered [ 2424.494212] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 06:02:11 (1789380131) [ 2425.539288] Key type .llcrypt unregistered [ 2425.546662] Key type ._llcrypt unregistered [ 2436.276809] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2436.280633] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2439.101978] Key type ._llcrypt registered [ 2439.104492] Key type .llcrypt registered [ 2439.277726] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2439.292862] alg: No test for adler32 (adler32-zlib) [ 2440.631848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2440.765826] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2459.435838] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 06:02:46 (1789380166) [ 2459.815234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2459.835374] LNet: 107318:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2459.844290] LNetError: 107318:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2459.871439] LNet: Removed LNI 10.1.2.3@tcp [ 2459.903666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2460.640728] Key type .llcrypt unregistered [ 2460.642102] Key type ._llcrypt unregistered [ 2467.858369] Key type ._llcrypt registered [ 2467.860718] Key type .llcrypt registered [ 2468.120470] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2468.142316] alg: No test for adler32 (adler32-zlib) [ 2469.062584] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2469.132186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 2469.348621] LNet: 107604:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2469.367991] LNetError: 107604:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2469.391071] LNet: Removed LNI 192.168.202.10@tcp [ 2470.286736] Key type .llcrypt unregistered [ 2470.289286] Key type ._llcrypt unregistered [ 2487.195596] Key type ._llcrypt registered [ 2487.201205] Key type .llcrypt registered [ 2490.191176] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 06:03:16 (1789380196) [ 2491.891666] Key type .llcrypt unregistered [ 2491.901189] Key type ._llcrypt unregistered [ 2504.074618] Key type ._llcrypt registered [ 2504.077731] Key type .llcrypt registered [ 2504.497691] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2504.530251] alg: No test for adler32 (adler32-zlib) [ 2516.178825] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 06:03:42 (1789380222) [ 2516.384310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2516.462986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2516.630865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2516.678243] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2518.095405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2518.199344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2518.240326] LNet: Removed LNI 192.168.202.10@tcp [ 2518.708431] LNet: 109365:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2519.404365] Key type .llcrypt unregistered [ 2519.406341] Key type ._llcrypt unregistered [ 2534.741441] Key type ._llcrypt registered [ 2534.746516] Key type .llcrypt registered [ 2536.614168] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 06:04:03 (1789380243) [ 2537.243763] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2537.272354] alg: No test for adler32 (adler32-zlib) [ 2538.315701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2538.398584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2538.438980] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2539.546839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 2539.673329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.202.10@tcp [ 2539.712357] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2539.718568] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2539.723538] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2539, 0, 0 [ 2539.728085] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2539.734371] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2539.737736] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2539.749171] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2539, 0, 0 [ 2539.760345] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2539.771400] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2539.780127] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2539.788424] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2539, 0, 2589 [ 2539.795552] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2539.800383] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2539.814426] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2539.822646] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2539, 0, 2589 [ 2539.831300] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2548.595774] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 06:04:16 (1789380256) [ 2549.565241] LNet: 110926:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2549.577796] LNetError: 110926:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2549.611280] LNet: Removed LNI 192.168.202.10@tcp [ 2550.661429] Key type .llcrypt unregistered [ 2550.663878] Key type ._llcrypt unregistered [ 2558.644637] Key type ._llcrypt registered [ 2558.646729] Key type .llcrypt registered [ 2558.880345] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2558.904174] alg: No test for adler32 (adler32-zlib) [ 2559.912153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2559.937226] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2571.712634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.110@tcp [ 2584.970183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2585.015793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2585.057962] LNet: Added LNI 192.168.202.10@tcp1 [8/256/0/180] [ 2586.722165] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2586.994407] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2587.009856] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2587.022809] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2589.767400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2589.812961] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2606.432932] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 06:05:13 (1789380313) [ 2606.783972] LNet: 112440:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2606.816072] LNetError: 112440:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2606.850093] LNet: Removed LNI 192.168.202.10@tcp [ 2606.858286] LNet: Removed LNI 192.168.202.10@tcp1 [ 2606.957680] LNet: Removed LNI 10.1.2.3@tcp99 [ 2608.960777] Key type .llcrypt unregistered [ 2608.971154] Key type ._llcrypt unregistered [ 2621.872819] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2621.881810] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2623.759309] Key type ._llcrypt registered [ 2623.761607] Key type .llcrypt registered [ 2624.051785] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2624.086488] alg: No test for adler32 (adler32-zlib) [ 2624.921218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2624.958371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2624.978106] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2625.538524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2625.547579] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2628.512914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2628.607943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2628.684887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2628.733736] LNetError: 113178:0:(peer.c:2366:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2628.799909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2628.859958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 2629.267990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2629.405813] LNet: 113198:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2629.411983] LNetError: 113198:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2629.416970] LNet: Removed LNI 10.1.2.3@tcp [ 2630.440132] LNet: Removed LNI 192.168.202.10@tcp [ 2631.589567] Key type .llcrypt unregistered [ 2631.591247] Key type ._llcrypt unregistered [ 2648.326522] Key type ._llcrypt registered [ 2648.329475] Key type .llcrypt registered [ 2650.382221] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 06:05:57 (1789380357) [ 2650.925869] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2650.955749] alg: No test for adler32 (adler32-zlib) [ 2651.994571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2652.102750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2652.137410] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2653.254848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 2653.462196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg210-client@tcp [ 2661.797978] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 06:06:09 (1789380369) [ 2662.203711] LNet: 114747:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2662.208464] LNetError: 114747:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2662.244059] LNet: Removed LNI 192.168.202.10@tcp [ 2662.935483] Key type .llcrypt unregistered [ 2662.938257] Key type ._llcrypt unregistered [ 2670.767813] Key type ._llcrypt registered [ 2670.770601] Key type .llcrypt registered [ 2670.960139] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2670.983988] alg: No test for adler32 (adler32-zlib) [ 2671.939709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2671.980138] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2672.012805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2680.515842] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 06:06:27 (1789380387) [ 2682.788086] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2685.720036] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 06:06:32 (1789380392) [ 2687.786974] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2690.846773] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 06:06:37 (1789380397) [ 2690.936910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2690.960906] LNet: 116179:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2690.973339] LNetError: 116179:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2690.993351] LNet: Removed LNI 192.168.202.10@tcp [ 2691.049700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2691.156151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2691.274765] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2691.849321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2691.905303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2698.847114] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 06:06:46 (1789380406) [ 2698.987689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2699.025584] LNet: 116901:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2699.036127] LNetError: 116901:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2699.076201] LNet: Removed LNI 192.168.202.10@tcp [ 2699.114517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2699.157550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2699.180530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2699.851603] Key type .llcrypt unregistered [ 2699.853987] Key type ._llcrypt unregistered [ 2714.094799] Key type ._llcrypt registered [ 2714.097502] Key type .llcrypt registered [ 2715.581625] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 06:07:03 (1789380423) [ 2716.042028] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2716.049586] alg: No test for adler32 (adler32-zlib) [ 2717.077802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2717.156799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2717.238889] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2717.891759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2717.931514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 2725.936976] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 06:07:13 (1789380433) [ 2726.094852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2726.127729] LNet: 118468:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2726.145098] LNetError: 118468:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2726.184112] LNet: Removed LNI 192.168.202.10@tcp [ 2726.280306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2726.482377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2726.552861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2735.171769] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 06:07:22 (1789380442) [ 2735.608429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2735.664657] LNet: 119066:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2735.711715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2735.834722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2736.525932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2736.600067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2736.744578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2737.430222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2737.487153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2737.636391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2737.684798] LNet: 119099:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2737.760096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2737.880342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2738.638694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2738.715340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2738.860619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2739.492627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2739.557490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2747.971048] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 06:07:36 (1789380456) [ 2748.437741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2748.473126] LNet: 119710:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2748.490618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2748.531270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2748.544410] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2749.084547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2749.113363] LNet: Added LNI 192.168.202.10@tcp2 [8/256/0/180] [ 2750.544271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2750.597896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 2750.821231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2750.843206] LNetError: 119989:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2751.903178] LNet: Removed LNI 192.168.202.10@tcp [ 2751.905251] LNetError: 119989:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2751.915042] LNet: Removed LNI 192.168.202.10@tcp2 [ 2751.926098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2751.959204] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2751.963941] LNet: Added LNI 192.168.202.10@tcp2 [8/256/0/180] [ 2751.969380] LNet: 119995:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2752.360746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 2752.562810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2752.593191] LNet: 120032:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2752.606388] LNet: 120032:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2753.636414] LNet: Removed LNI 192.168.202.10@tcp [ 2753.660895] LNet: Removed LNI 192.168.202.10@tcp2 [ 2753.699684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2753.758828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2753.774150] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2754.305373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2754.336879] LNet: Added LNI 192.168.202.10@tcp2 [8/256/0/180] [ 2755.334425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 2755.364316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2755.412979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 2755.635125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2755.653358] LNetError: 120317:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2755.658047] LNetError: 120317:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2755.663718] LNet: Removed LNI 192.168.202.10@tcp [ 2756.712634] LNet: Removed LNI 192.168.202.10@tcp2 [ 2756.747923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2756.879743] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2756.884247] LNet: Added LNI 192.168.202.10@tcp2 [8/256/0/180] [ 2756.887737] LNet: 120323:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2757.150086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 2757.408850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2757.468855] LNet: Removed LNI 192.168.202.10@tcp [ 2758.507933] LNet: Removed LNI 192.168.202.10@tcp2 [ 2758.540534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2758.679627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2758.736591] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2759.561833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2759.580840] LNet: Added LNI 192.168.202.10@tcp2 [8/256/0/180] [ 2760.879923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2760.999793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 2761.403964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2761.420635] LNet: 120640:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2761.424361] LNet: 120640:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2761.437486] LNetError: 120640:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2761.439881] LNetError: 120640:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2761.456379] LNet: Removed LNI 192.168.202.10@tcp [ 2761.476427] LNet: Removed LNI 192.168.202.10@tcp2 [ 2761.504836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2761.611683] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2761.616378] LNet: Added LNI 192.168.202.10@tcp2 [8/256/0/180] [ 2761.623981] LNet: 120646:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2762.316514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 2762.532376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2763.615122] LNet: Removed LNI 192.168.202.10@tcp [ 2763.654912] LNet: Removed LNI 192.168.202.10@tcp2 [ 2763.709073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2763.795481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2763.863472] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2765.125745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2765.154753] LNet: Added LNI 192.168.202.10@tcp2 [8/256/0/180] [ 2766.914071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 2766.994817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2767.075931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 2767.391400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2768.479301] LNet: Removed LNI 192.168.202.10@tcp [ 2769.511522] LNet: Removed LNI 192.168.202.10@tcp2 [ 2769.555205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2769.703679] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2769.715333] LNet: Added LNI 192.168.202.10@tcp2 [8/256/0/180] [ 2769.724186] LNet: 120974:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2770.355032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 2770.521139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2770.540742] LNetError: 121011:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2770.546964] LNetError: 121011:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2770.556696] LNet: Removed LNI 192.168.202.10@tcp [ 2771.623841] LNet: Removed LNI 192.168.202.10@tcp2 [ 2771.681212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2771.746851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2771.800836] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2772.728251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2772.740969] LNet: Added LNI 192.168.202.10@tcp2 [8/256/0/180] [ 2773.614813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2773.653757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2773.701680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 2774.025340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2775.072457] LNet: Removed LNI 192.168.202.10@tcp [ 2775.089251] LNet: Removed LNI 192.168.202.10@tcp2 [ 2775.130475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2775.320944] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2775.337664] LNet: Added LNI 192.168.202.10@tcp2 [8/256/0/180] [ 2775.355933] LNet: 121323:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2776.320462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 2776.747780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2777.823224] LNet: Removed LNI 192.168.202.10@tcp [ 2777.850614] LNet: Removed LNI 192.168.202.10@tcp2 [ 2777.892740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2777.984464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2778.039695] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2779.012764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2779.028458] LNet: Added LNI 192.168.202.10@tcp2 [8/256/0/180] [ 2780.156709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 2780.201205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2780.242880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2780.324535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 2780.766084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2780.814601] LNet: 121677:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2780.817321] LNet: 121677:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 2781.855212] LNet: Removed LNI 192.168.202.10@tcp [ 2782.887187] LNet: Removed LNI 192.168.202.10@tcp2 [ 2782.938975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2783.081150] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2783.098856] LNet: Added LNI 192.168.202.10@tcp2 [8/256/0/180] [ 2783.990206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 2791.980518] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2793.708041] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 06:08:21 (1789380501) [ 2793.920387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2793.972400] LNetError: 122451:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2793.982327] LNetError: 122451:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2793.989191] LNet: Removed LNI 192.168.202.10@tcp [ 2795.049897] LNet: Removed LNI 192.168.202.10@tcp2 [ 2795.087763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2795.543436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2795.561362] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2796.357392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2796.376474] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2797.337587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2797.364736] LNet: Removed LNI 10.1.2.3@tcp [ 2797.378191] LNet: Removed LNI 192.168.202.10@tcp [ 2797.420604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2797.547471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2797.573783] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2797.580404] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2798.081420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2798.138116] LNet: Removed LNI 192.168.202.10@tcp [ 2798.152894] LNet: Removed LNI 10.1.2.3@tcp [ 2806.464467] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 06:08:33 (1789380513) [ 2806.675929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2806.769219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2806.842881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2806.867671] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2808.695209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2808.763253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2808.894266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 2809.013048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 2809.073382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 2809.130622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 2809.212200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 2809.267548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 2809.394077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 2809.481962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 2809.565684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 2809.678505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 2810.200685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2822.078985] Lustre: DEBUG MARKER: == sanity-lnet test 476: lnetctl set latency_stats toggles capture ========================================================== 06:08:49 (1789380529) [ 2822.788473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2822.817611] LNet: 124087:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2822.831355] LNet: 124087:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2823.910796] LNet: Removed LNI 192.168.202.10@tcp [ 2823.966692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2824.025137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2824.099401] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2824.818953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 0 [ 2824.855517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2824.904286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 2824.934481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 2824.972711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 2825.010188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 2825.039559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 2825.074727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 2825.106942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 2825.140828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 2825.169988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 2825.194989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 2825.265757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2825.314748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 2825.349925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 2825.390048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 2825.414917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 2825.441941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 2825.475383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 2825.503268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 2825.536057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 2825.562975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 2825.598250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 2834.116346] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 06:09:01 (1789380541) [ 2834.494461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2834.541843] LNetError: 124852:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2834.554431] LNetError: 124852:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 2834.593274] LNet: Removed LNI 192.168.202.10@tcp [ 2834.642819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2834.846259] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2835.100963] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2835.169925] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2835.172643] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2837.729087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2837.785716] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2838.993384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2839.029872] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2841.671741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 2841.767695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2841.925668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.202.10@tcp -m PUT -e local_timeout -r 1 [ 2842.080530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2842.189641] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2842.214203] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2842.564180] LNetError: 124917:0:(lib-move.c:4794:lnet_parse()) 192.168.202.10@tcp, src 192.168.202.10@tcp: Dropping PUT (error -108 looking up sender) [ 2843.615950] LNet: Removed LNI 10.1.2.3@tcp [ 2844.648583] LNet: Removed LNI 192.168.202.10@tcp [ 2845.206560] Key type .llcrypt unregistered [ 2845.213974] Key type ._llcrypt unregistered [ 2861.517385] Key type ._llcrypt registered [ 2861.522315] Key type .llcrypt registered [ 2863.429359] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 06:09:30 (1789380570) [ 2863.985746] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2864.032828] alg: No test for adler32 (adler32-zlib) [ 2865.098301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2865.307522] LNet: 126242:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2865.712370] Key type .llcrypt unregistered [ 2865.714332] Key type ._llcrypt unregistered [ 2872.391612] Key type ._llcrypt registered [ 2872.396681] Key type .llcrypt registered [ 2872.676956] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2872.697230] alg: No test for adler32 (adler32-zlib) [ 2873.691266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2873.709595] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2881.592703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.110@tcp [ 2881.893867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2882.706915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.110@tcp [ 2882.917225] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.110@tcp [ 2882.999846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2883.040715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2883.093542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2883.143877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2883.203874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2883.244253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2883.300066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2883.358935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2883.417488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2883.449358] LNet: 126693:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2883.468704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2883.507473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2883.559843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2883.620324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2883.648806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2883.676745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2884.314251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2884.575282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2884.602433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2884.635593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2884.688708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2884.743431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2884.809989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2884.873597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2884.923557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2884.961646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2884.978840] LNet: 126812:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2885.008699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2885.052361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2885.079809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2885.107890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2885.145596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2885.176360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2885.733580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.110@tcp [ 2886.045470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2886.075941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2886.135694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2886.173892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2886.230550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2886.274789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2886.328689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2886.381931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2886.404909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2886.419575] LNet: 126932:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2886.433401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2886.458867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2886.488891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2886.520504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2886.573939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2886.621488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2887.297259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2887.563778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2887.600370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2887.639693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2887.672566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2887.700814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2887.727841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2887.753526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2887.776331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2887.805488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2887.838365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2887.866817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2887.896794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2887.923260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2887.949021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2887.975727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2888.274730] LNet: 127097:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2888.282881] LNetError: 127097:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2889.316901] LNet: Removed LNI 192.168.202.10@tcp [ 2889.673079] Key type .llcrypt unregistered [ 2889.678427] Key type ._llcrypt unregistered [ 2907.037533] Key type ._llcrypt registered [ 2907.040174] Key type .llcrypt registered [ 2908.278287] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 06:10:16 (1789380616) [ 2908.993165] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2909.022296] alg: No test for adler32 (adler32-zlib) [ 2910.075786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2910.635544] LNet: 127994:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2911.681392] Key type .llcrypt unregistered [ 2911.684745] Key type ._llcrypt unregistered [ 2920.712338] Key type ._llcrypt registered [ 2920.716355] Key type .llcrypt registered [ 2921.138667] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2921.176144] alg: No test for adler32 (adler32-zlib) [ 2922.250590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2922.278510] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2931.655304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.110@tcp [ 2941.307817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2941.362685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2941.402490] LNet: Added LNI 192.168.202.10@tcp1 [8/256/0/180] [ 2942.331362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2943.084559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.110@tcp [ 2943.334949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2943.353752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2943.375703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2943.394705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2943.429940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2943.451284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2943.467692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2943.484681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2943.525642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2943.562966] LNet: 128650:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2943.586910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2943.638780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2943.693950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2943.731777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2943.775187] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.110@tcp [ 2943.799688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2943.863349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2944.927417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2945.298431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2945.322213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2945.344309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2945.371239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2945.395316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2945.417769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2945.442197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2945.471269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2945.505589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2945.520867] LNet: 128769:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2945.536864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2945.560422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2945.609580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2945.661966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2945.707718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2945.753236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2946.682132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.110@tcp [ 2947.042674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2947.117444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2947.180640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2947.247595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2947.305380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2947.383574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2947.432796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2947.463520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2947.493546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2947.518409] LNet: 128889:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2947.538216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2947.571639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2947.607933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2947.642802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2947.670372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2947.706100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2948.736672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2949.033683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2949.065959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2949.103674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2949.133762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2949.164159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2949.195658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2949.228408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2949.258454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2949.296195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2949.331275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2949.385198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2949.426264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2949.454970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2949.482412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2949.528304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2955.226514] LNet: 129089:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2955.257605] LNetError: 129089:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2955.269911] LNet: Removed LNI 192.168.202.10@tcp [ 2955.301431] LNet: Removed LNI 192.168.202.10@tcp1 [ 2955.841788] Key type .llcrypt unregistered [ 2955.848572] Key type ._llcrypt unregistered [ 2974.568611] Key type ._llcrypt registered [ 2974.570202] Key type .llcrypt registered [ 2976.822223] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 06:11:23 (1789380683) [ 2977.273044] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2977.298799] alg: No test for adler32 (adler32-zlib) [ 2978.152630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2978.203831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2978.230306] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2978.990330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2979.007344] LNet: Added LNI 192.168.202.10@tcp2 [8/256/0/180] [ 2980.069877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 2980.262068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2981.343749] LNet: Removed LNI 192.168.202.10@tcp [ 2993.868476] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 06:11:41 (1789380701) [ 2996.544811] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2999.628138] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 06:11:46 (1789380706) [ 3011.080655] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 06:11:58 (1789380718) [ 3020.435045] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 06:12:08 (1789380728) [ 3030.405598] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 06:12:17 (1789380737) [ 3040.630665] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 06:12:27 (1789380747) [ 3050.833698] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 06:12:38 (1789380758) [ 3061.511665] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 06:12:48 (1789380768) [ 3072.270238] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 06:12:59 (1789380779) [ 3082.600669] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 06:13:10 (1789380790) [ 3093.606895] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 06:13:20 (1789380800) [ 3105.492120] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 06:13:32 (1789380812) [ 3115.663207] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 06:13:43 (1789380823) [ 3123.344766] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 06:13:51 (1789380831) [ 3133.192909] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 06:14:00 (1789380840) [ 3142.527837] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 06:14:09 (1789380849) [ 3152.617908] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 06:14:20 (1789380860) [ 3165.280716] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 06:14:32 (1789380872) [ 3178.498090] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 06:14:45 (1789380885) [ 3179.268649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3179.327750] LNet: 140954:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3179.376529] LNetError: 140954:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3179.425167] LNet: Removed LNI 192.168.202.10@tcp2 [ 3179.493452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3179.853191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3179.900099] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 3181.538709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3181.572941] LNet: 141096:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3181.595679] LNetError: 141096:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3181.609822] LNet: Removed LNI 192.168.202.10@tcp [ 3181.633924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3181.679969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 3181.699898] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 3190.668760] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 06:14:58 (1789380898) [ 3199.690982] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 06:15:07 (1789380907) [ 3208.138412] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 06:15:15 (1789380915) [ 3219.650675] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 06:15:26 (1789380926) [ 3230.973939] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 06:15:37 (1789380937) [ 3241.595513] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 06:15:49 (1789380949) [ 3251.279542] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 06:15:59 (1789380959) [ 3258.989272] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 06:16:06 (1789380966) [ 3266.436825] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 06:16:14 (1789380974) [ 3278.396344] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 06:16:25 (1789380985) [ 3290.050708] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 06:16:38 (1789380998) [ 3298.810949] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 06:16:47 (1789381007) [ 3306.489743] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 3010 sec ========== 06:16:54 (1789381014) [ 3306.756630] LNet: 148671:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3306.768613] LNetError: 148671:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3306.784892] LNet: Removed LNI 192.168.202.10@tcp [ 3307.746829] Key type .llcrypt unregistered [ 3307.748527] Key type ._llcrypt unregistered [ 3323.397218] Key type ._llcrypt registered [ 3323.405696] Key type .llcrypt registered [ 3323.951473] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 3323.980441] alg: No test for adler32 (adler32-zlib) [ 3325.189382] Lustre: Lustre: Build Version: 2.17.58_2_g65a210d [ 3325.625448] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 3327.351205] Key type lgssc registered [ 3328.969913] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3449.886964] Lustre: Mounted lustre-client - version 2.17.58_2_g65a210d [ 3457.496384] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3468.631472] Lustre: Unmounted lustre-client [ 3520.718479] Key type lgssc unregistered [ 3521.093057] LNet: 152185:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3521.118659] LNetError: 152185:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3522.157805] LNet: Removed LNI 192.168.202.10@tcp [ 3523.616495] Key type .llcrypt unregistered [ 3523.617904] Key type ._llcrypt unregistered