[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 421742071 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.003221] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005017] kvm-guest: setup PV IPIs [ 0.008649] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010013] pid_max: default: 32768 minimum: 301 [ 0.011136] LSM: Security Framework initializing [ 0.012059] Yama: becoming mindful. [ 0.013037] SELinux: Initializing. [ 0.014071] *** VALIDATE selinux *** [ 0.023448] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027809] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028171] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029108] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030126] *** VALIDATE tmpfs *** [ 0.031479] *** VALIDATE proc *** [ 0.032235] *** VALIDATE cgroup *** [ 0.033009] *** VALIDATE cgroup2 *** [ 0.035141] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037063] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039028] Spectre V2 : User space: Vulnerable [ 0.040010] Speculative Store Bypass: Vulnerable [ 0.042669] debug: unmapping init [mem 0xffffffff94a59000-0xffffffff94a60fff] [ 0.045180] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046682] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047027] ... version: 2 [ 0.048016] ... bit width: 48 [ 0.049014] ... generic registers: 4 [ 0.050018] ... value mask: 0000ffffffffffff [ 0.051016] ... max period: 00007fffffffffff [ 0.052016] ... fixed-purpose events: 3 [ 0.053011] ... event mask: 000000070000000f [ 0.054349] rcu: Hierarchical SRCU implementation. [ 0.056454] smp: Bringing up secondary CPUs ... [ 0.057588] x86: Booting SMP configuration: [ 0.058025] .... node #0, CPUs: #1 #2 #3 [ 0.061360] smp: Brought up 1 node, 4 CPUs [ 0.063013] smpboot: Max logical packages: 1 [ 0.064018] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.137611] node 0 deferred pages initialised in 71ms [ 0.140608] devtmpfs: initialized [ 0.142169] x86/mm: Memory block size: 128MB [ 0.144863] gcov: version magic: 0x41383552 [ 0.145639] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.149100] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.152327] pinctrl core: initialized pinctrl subsystem [ 0.153163] [ 0.153582] ************************************************************* [ 0.156010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.157009] ** ** [ 0.159011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.161015] ** ** [ 0.163011] ** This means that this kernel is built to expose internal ** [ 0.166024] ** IOMMU data structures, which may compromise security on ** [ 0.168015] ** your system. ** [ 0.170064] ** ** [ 0.173013] ** If you see this message and you are not debugging the ** [ 0.175011] ** kernel, report this immediately to your vendor! ** [ 0.177014] ** ** [ 0.179012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.181017] ************************************************************* [ 0.184594] NET: Registered protocol family 16 [ 0.186444] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.189065] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.192064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.196116] cpuidle: using governor menu [ 0.197599] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.198472] PCI: Using configuration type 1 for base access [ 0.199124] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.206195] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.208062] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.211320] cryptd: max_cpu_qlen set to 1000 [ 0.214516] ACPI: Added _OSI(Module Device) [ 0.216022] ACPI: Added _OSI(Processor Device) [ 0.217021] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.219014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.222784] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.227421] ACPI: Interpreter enabled [ 0.229057] ACPI: PM: (supports S0 S3 S4 S5) [ 0.230019] ACPI: Using IOAPIC for interrupt routing [ 0.231098] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.234283] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.242417] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.245046] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.247014] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.249064] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.252013] acpiphp: Slot [2] registered [ 0.253052] acpiphp: Slot [5] registered [ 0.253990] acpiphp: Slot [6] registered [ 0.255096] acpiphp: Slot [3] registered [ 0.255971] acpiphp: Slot [4] registered [ 0.257068] acpiphp: Slot [7] registered [ 0.258096] acpiphp: Slot [8] registered [ 0.259070] acpiphp: Slot [9] registered [ 0.260102] acpiphp: Slot [10] registered [ 0.260989] acpiphp: Slot [11] registered [ 0.262083] acpiphp: Slot [12] registered [ 0.263091] acpiphp: Slot [13] registered [ 0.265114] acpiphp: Slot [14] registered [ 0.266063] acpiphp: Slot [15] registered [ 0.266954] acpiphp: Slot [16] registered [ 0.268078] acpiphp: Slot [17] registered [ 0.269204] acpiphp: Slot [18] registered [ 0.270111] acpiphp: Slot [19] registered [ 0.272109] acpiphp: Slot [20] registered [ 0.273132] acpiphp: Slot [21] registered [ 0.275107] acpiphp: Slot [22] registered [ 0.276092] acpiphp: Slot [23] registered [ 0.277074] acpiphp: Slot [24] registered [ 0.278072] acpiphp: Slot [25] registered [ 0.279062] acpiphp: Slot [26] registered [ 0.280108] acpiphp: Slot [27] registered [ 0.281122] acpiphp: Slot [28] registered [ 0.281999] acpiphp: Slot [29] registered [ 0.283064] acpiphp: Slot [30] registered [ 0.284027] acpiphp: Slot [31] registered [ 0.284967] PCI host bridge to bus 0000:00 [ 0.286024] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.288023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.290019] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.292044] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.294023] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.297027] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.299186] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.302099] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.306000] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.314612] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.319061] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.322026] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.324018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.326019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.328536] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.331810] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.334039] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.336638] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.339014] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.348014] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.353016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.358000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.363012] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.368017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.382015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.390976] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.404032] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.411017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.431022] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.448320] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.451411] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.453328] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.455439] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.457231] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.462140] iommu: Default domain type: Passthrough [ 0.464460] SCSI subsystem initialized [ 0.465121] ACPI: bus type USB registered [ 0.467088] usbcore: registered new interface driver usbfs [ 0.469072] usbcore: registered new interface driver hub [ 0.470075] usbcore: registered new device driver usb [ 0.471120] pps_core: LinuxPPS API ver. 1 registered [ 0.473012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.475059] PTP clock support registered [ 0.477071] EDAC MC: Ver: 3.0.0 [ 0.479115] PCI: Using ACPI for IRQ routing [ 0.480547] NetLabel: Initializing [ 0.481016] NetLabel: domain hash size = 128 [ 0.483011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.486109] NetLabel: unlabeled traffic allowed by default [ 0.488128] vgaarb: loaded [ 0.489271] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.491022] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.497463] clocksource: Switched to clocksource kvm-clock [ 0.605751] VFS: Disk quotas dquot_6.6.0 [ 0.606946] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.609486] *** VALIDATE ramfs *** [ 0.610818] *** VALIDATE hugetlbfs *** [ 0.612866] pnp: PnP ACPI init [ 0.615298] pnp: PnP ACPI: found 6 devices [ 0.635633] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.637776] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.639249] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.640628] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.642368] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.644498] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.647048] NET: Registered protocol family 2 [ 0.649175] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.652553] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.655619] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.660882] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.663946] TCP: Hash tables configured (established 65536 bind 65536) [ 0.666018] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.668446] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.670346] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.672313] NET: Registered protocol family 1 [ 0.674179] RPC: Registered named UNIX socket transport module. [ 0.676533] RPC: Registered udp transport module. [ 0.677631] RPC: Registered tcp transport module. [ 0.679095] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.680961] NET: Registered protocol family 44 [ 0.682054] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.683943] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.686065] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.687792] PCI: CLS 0 bytes, default 64 [ 0.689151] Unpacking initramfs... [ 2.037567] debug: unmapping init [mem 0xffff92b4bcc64000-0xffff92b4bffcffff] [ 2.043553] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.045148] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.048492] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.518222] Initialise system trusted keyrings [ 2.519506] Key type blacklist registered [ 2.522053] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.532189] zbud: loaded [ 2.535228] *** VALIDATE nfs *** [ 2.536129] *** VALIDATE nfs4 *** [ 2.537369] pstore: using deflate compression [ 2.540408] Platform Keyring initialized [ 2.652546] NET: Registered protocol family 38 [ 2.653864] Key type asymmetric registered [ 2.654881] Asymmetric key parser 'x509' registered [ 2.656295] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.658432] io scheduler mq-deadline registered [ 2.659660] io scheduler kyber registered [ 2.661231] io scheduler bfq registered [ 2.662749] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.664840] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.666944] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.668812] ACPI: Power Button [PWRF] [ 2.673782] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.679511] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.688786] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.715689] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.743225] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.747901] Non-volatile memory driver v1.3 [ 2.748716] Linux agpgart interface v0.103 [ 2.781479] virtio_blk virtio1: [vda] 145896 512-byte logical blocks (74.7 MB/71.2 MiB) [ 2.783920] vda: detected capacity change from 0 to 74698752 [ 2.799902] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.802363] vdb: detected capacity change from 0 to 1073741824 [ 2.808929] libphy: Fixed MDIO Bus: probed [ 2.814247] usbcore: registered new interface driver usbserial_generic [ 2.816156] usbserial: USB Serial support registered for generic [ 2.817569] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.820485] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.821891] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.823575] mousedev: PS/2 mouse device common for all mice [ 2.826457] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.828644] rtc_cmos 00:05: RTC can wake from S4 [ 2.833175] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.833290] rtc_cmos 00:05: registered as rtc0 [ 2.837375] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.839962] intel_pstate: CPU model not supported [ 2.841857] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.844508] hid: raw HID events driver (C) Jiri Kosina [ 2.846704] usbcore: registered new interface driver usbhid [ 2.847992] usbhid: USB HID core driver [ 2.849082] drop_monitor: Initializing network drop monitor service [ 2.850977] Initializing XFRM netlink socket [ 2.853326] NET: Registered protocol family 10 [ 2.856625] Segment Routing with IPv6 [ 2.857605] NET: Registered protocol family 17 [ 2.859431] mpls_gso: MPLS GSO support [ 2.865379] RAS: Correctable Errors collector initialized. [ 2.867549] AVX version of gcm_enc/dec engaged. [ 2.869157] AES CTR mode by8 optimization enabled [ 2.948397] sched_clock: Marking stable (2948356009, 0)->(3844513101, -896157092) [ 2.951805] registered taskstats version 1 [ 2.953610] Loading compiled-in X.509 certificates [ 2.955311] zswap: loaded using pool lzo/zbud [ 2.979753] Key type big_key registered [ 2.993267] Key type encrypted registered [ 2.994652] ima: No TPM chip found, activating TPM-bypass! [ 2.995895] ima: Allocated hash algorithm: sha1 [ 2.997418] ima: No architecture policies found [ 2.998354] evm: Initialising EVM extended attributes: [ 2.999203] evm: security.selinux [ 2.999915] evm: security.ima [ 3.000387] evm: security.capability [ 3.000977] evm: HMAC attrs: 0x1 [ 3.002370] rtc_cmos 00:05: setting system clock to 2026-08-08 11:56:34 UTC (1786190194) [ 3.007159] debug: unmapping init [mem 0xffffffff95a03000-0xffffffff95bfffff] [ 3.009046] debug: unmapping init [mem 0xffffffff94782000-0xffffffff94a58fff] [ 3.017111] Write protecting the kernel read-only data: 28672k [ 3.020393] debug: unmapping init [mem 0xffffffff92e03000-0xffffffff92ffffff] [ 3.022772] debug: unmapping init [mem 0xffffffff93714000-0xffffffff937fffff] [ 3.049955] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.055460] systemd[1]: Detected virtualization kvm. [ 3.056441] systemd[1]: Detected architecture x86-64. [ 3.057372] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.083962] systemd[1]: No hostname configured. [ 3.085679] systemd[1]: Set hostname to . [ 3.087302] random: systemd: uninitialized urandom read (16 bytes read) [ 3.089073] systemd[1]: Initializing machine ID from random generator. [ 3.117131] random: ln: uninitialized urandom read (6 bytes read) [ 3.219266] random: systemd: uninitialized urandom read (16 bytes read) [ 3.221832] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.226377] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.230113] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local File Systems. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.884555] device-mapper: uevent: version 1.0.3 [ 3.886777] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ 4.258612] random: fast init done [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.740291] virtio_net virtio0 ens2: renamed from eth0 [ 4.831880] scsi host0: ata_piix [ 4.874254] scsi host1: ata_piix [ 4.875578] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.877385] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.692396] dracut-initqueue[596]: RTNETLINK answers: File exists [ 9.505786] random: crng init done [ 9.507220] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.131091] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.207462] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.471599] SELinux: Disabled at runtime. [ 11.534369] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.542356] systemd[1]: Detected virtualization kvm. [ 11.544544] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.007965] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.011603] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.016513] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.020318] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.024120] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.032114] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.036519] systemd[1]: Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Kernel Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice User and Session Slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ 12.076203] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Create list of required st…ce nodes for the current kernel... Mounting Huge Pages File System... Mounting Kernel Debug File System... Starting Remount Root and Kernel File Systems... [ OK ] Listening on initctl Compatibility Named Pipe. Mounting POSIX Message Queue File System... [ OK ] Reached target Paths. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Starting Apply Kernel Variables... [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 12.496230] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.802454] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.873395] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.880870] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.921086] EDAC sbridge: Ver: 1.1.2 [ 14.047518] Key type dns_resolver registered [ 14.363479] NFS: Registering the id_resolver key type [ 14.365158] Key type id_resolver registered [ 14.366674] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started D-Bus System Message Bus. [ OK ] Reached target sshd-keygen.target. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg220-client login: [ 44.348451] libcfs: loading out-of-tree module taints kernel. [ 44.459723] Key type ._llcrypt registered [ 44.478735] Key type .llcrypt registered [ 44.821888] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 44.829606] alg: No test for adler32 (adler32-zlib) [ 45.911501] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 46.288723] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 47.943272] Key type lgssc registered [ 48.767096] Lustre: Echo OBD driver; http://www.lustre.org/ [ 119.072017] hrtimer: interrupt took 5991018 ns [ 202.574206] Lustre: Mounted lustre-client - version 2.17.56_51_geba2e8d [ 208.867194] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 225.468710] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing check_logdir /tmp/testlogs/ [ 228.350093] Lustre: lustre-OST0000-osc-ffff92b511331000: disconnect after 23s idle [ 232.321625] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing yml_node [ 237.971858] Lustre: DEBUG MARKER: Client: 2.17.56.51 [ 240.687959] Lustre: DEBUG MARKER: MDS: 2.17.56.51 [ 243.837265] Lustre: DEBUG MARKER: OSS: 2.17.56.51 [ 245.837899] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Sat Aug 8 08:00:35 EDT 2026 [ 267.431127] Lustre: Unmounted lustre-client [ 297.725418] Key type lgssc unregistered [ 298.113842] LNet: 8446:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 298.125362] LNetError: 8446:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 298.153357] LNet: Removed LNI 192.168.202.20@tcp [ 299.193508] Key type .llcrypt unregistered [ 299.197729] Key type ._llcrypt unregistered [ 332.945575] Key type ._llcrypt registered [ 332.947726] Key type .llcrypt registered [ 333.430786] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 333.473293] alg: No test for adler32 (adler32-zlib) [ 334.531222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 334.660772] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 334.698601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 339.983668] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 340.297957] LNet: 9338:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 340.310990] LNetError: 9338:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 340.330880] LNet: Removed LNI 192.168.202.20@tcp [ 340.981379] Key type .llcrypt unregistered [ 340.986049] Key type ._llcrypt unregistered [ 349.869522] Key type ._llcrypt registered [ 349.884951] Key type .llcrypt registered [ 351.438471] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 08:02:21 (1786190541) [ 351.913896] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 351.945777] alg: No test for adler32 (adler32-zlib) [ 352.864902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 352.970295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 353.035559] LNet: 9753:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 353.082775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 353.179838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 353.237954] LNet: 9760:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 361.769565] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 08:02:31 (1786190551) [ 362.063153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 362.084251] LNet: 10339:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 362.121665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 362.233672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 371.110498] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 08:02:40 (1786190560) [ 371.229150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 371.265890] LNet: 10917:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 371.294835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 371.454208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 381.374774] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 08:02:50 (1786190570) [ 381.521318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 381.566567] LNet: 11501:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 381.614144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 381.733983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 389.476275] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 08:02:59 (1786190579) [ 389.830208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 389.865046] LNet: 12095:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 389.901065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 390.062683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 390.650664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 400.074582] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 08:03:09 (1786190589) [ 400.187028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 400.224427] LNet: 12678:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 400.267949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 400.420369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 401.188751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 401.857302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 410.888994] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 08:03:20 (1786190600) [ 411.137933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 411.208395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 411.317910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 419.790726] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 08:03:29 (1786190609) [ 420.400945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 420.439529] LNet: 13868:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 420.447438] LNet: 13868:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 420.477550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 420.591247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 420.636329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 421.289869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 421.328600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 421.966755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 422.007982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 422.717099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 422.777147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 423.649500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 423.722771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 424.065578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 424.098431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 424.603959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 424.646806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 425.160810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 425.233995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 425.997618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 426.043054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 426.360689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 426.391237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 427.057920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 427.147690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 428.015199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 428.085483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 428.918526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 428.989489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 437.967666] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 08:03:47 (1786190627) [ 438.125317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 438.184948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 438.284097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 438.314564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 447.629075] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 08:03:57 (1786190637) [ 448.061059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 448.128059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 448.210648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 448.263457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 458.140964] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 08:04:07 (1786190647) [ 458.316368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 458.369858] LNet: 15717:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 458.394944] LNet: 15717:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 458.441228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 458.584826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 458.622972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 468.287855] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 08:04:18 (1786190658) [ 468.360755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 468.397796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 468.477512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 468.546647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 477.803393] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 08:04:27 (1786190667) [ 477.976484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 478.068542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 478.231757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 478.278654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 488.682875] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 08:04:38 (1786190678) [ 488.931937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 489.010712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 489.186618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 489.239447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 499.408660] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 08:04:48 (1786190688) [ 499.558257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 499.623802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 499.730792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 500.518774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 509.388572] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 08:04:59 (1786190699) [ 509.855124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 509.920695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 510.214396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 510.635933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 520.390883] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 08:05:09 (1786190709) [ 521.070680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 521.189055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 521.373192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 521.668893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 529.122369] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 08:05:19 (1786190719) [ 529.429415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 529.459773] LNet: 19904:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 529.468951] LNet: 19904:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 6 previous similar messages [ 529.500902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 530.351269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 530.593385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 536.742534] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 08:05:26 (1786190726) [ 537.098569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 537.161544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 537.277567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 537.339938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 545.147758] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 08:05:35 (1786190735) [ 545.258022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 545.332117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 545.458349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 545.535833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 553.559802] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 08:05:43 (1786190743) [ 553.669366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 553.733117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 553.851445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 562.929920] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 08:05:52 (1786190752) [ 563.312478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 563.416784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 563.605709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 574.830660] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 08:06:03 (1786190763) [ 575.051617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 575.172922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 575.305627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 576.169993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 585.979576] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 08:06:14 (1786190774) [ 586.153818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 586.243821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 586.416477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 586.461670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 596.737924] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 08:06:26 (1786190786) [ 596.828932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 596.875063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 597.030864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 597.070944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 605.557787] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 08:06:35 (1786190795) [ 605.666406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 605.733200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 605.877966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 605.920644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 613.935530] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 08:06:43 (1786190803) [ 614.009762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 614.057366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 614.214593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 614.247890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 622.792868] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 08:06:52 (1786190812) [ 622.872755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 622.935941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 623.056952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 623.166261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 623.183306] LNetError: 25876:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 623.230559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 623.265581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 631.074706] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 08:07:00 (1786190820) [ 631.235962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 631.319923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 631.446677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.zGg0E2WVVd [ 631.503072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.zGg0E2WVVd [ 631.567894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.zGg0E2WVVd [ 639.013747] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 08:07:08 (1786190828) [ 639.155462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 639.222251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 639.304670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 639.390435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 639.480238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 647.237475] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 08:07:16 (1786190836) [ 647.327717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 647.364525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 647.449551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 647.473634] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 649.611710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.20@tcp [ 651.699687] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 653.791462] Key type lgssc registered [ 655.556772] Lustre: Echo OBD driver; http://www.lustre.org/ [ 660.156158] LNetError: 27638:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.202.20@tcp99 has no available nets [ 663.878669] Key type lgssc unregistered [ 664.354767] LNet: 28749:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 664.359907] LNet: 28749:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 13 previous similar messages [ 664.366817] LNetError: 28749:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 664.380458] LNet: Removed LNI 192.168.202.20@tcp [ 665.223822] Key type .llcrypt unregistered [ 665.225319] Key type ._llcrypt unregistered [ 675.116744] Key type ._llcrypt registered [ 675.119270] Key type .llcrypt registered [ 676.670924] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 08:07:46 (1786190866) [ 677.380787] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 677.393722] alg: No test for adler32 (adler32-zlib) [ 678.398981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 678.456488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 682.385217] LNet: Message forwarding will be enabled in 2s [ 685.471347] LNet: Message forwarding enabled [ 686.105661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 686.156739] LNet: Message forwarding will stop in 2s to 6s [ 689.705596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 690.119098] LNet: Message forwarding disabled [ 692.702427] LNet: Message forwarding will be enabled in 2s [ 696.288341] LNet: Message forwarding enabled [ 700.208882] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 08:08:10 (1786190890) [ 700.322275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 700.365444] LNet: 30144:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 701.879357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 702.010654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 709.152510] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 08:08:19 (1786190899) [ 709.310940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 709.334586] LNet: 30728:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 709.368728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 709.430973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 709.478592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 709.523531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 709.562889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 709.596543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 709.638875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 709.679665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 709.712876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 709.748764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 709.782417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 709.836626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 709.893396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 709.970390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 710.065757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 710.158629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 710.286721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 710.374973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 710.463632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 710.546578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 710.582473] LNetError: 30777:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 719.074533] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 08:08:28 (1786190908) [ 719.206824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 719.266461] LNet: 31340:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 719.313109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 719.453827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 728.860737] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 08:08:38 (1786190918) [ 731.950971] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg220-server [ 734.451611] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 08:08:43 (1786190923) [ 734.607247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 734.680175] LNet: 32275:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 734.710733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 734.823743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 734.895347] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 736.071556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.202.[21-25]@tcp [ 736.124738] LNet: 32415:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 736.182563] LNetError: 32284:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.21@tcp is being used as a gateway but routing feature is not turned on [ 736.219665] LNetError: 32284:0:(lib-lnet.h:1274:lnet_set_route_aliveness()) route to tcp1 through 192.168.202.21@tcp has gone from up to down [ 737.196758] LNetError: 32284:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.21@tcp is being used as a gateway but routing feature is not turned on [ 737.202958] LNetError: 32284:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 738.229931] LNetError: 32284:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.23@tcp is being used as a gateway but routing feature is not turned on [ 738.237783] LNetError: 32284:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 740.261747] LNetError: 32284:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.22@tcp is being used as a gateway but routing feature is not turned on [ 740.276882] LNetError: 32284:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 744.359272] LNetError: 32284:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.23@tcp is being used as a gateway but routing feature is not turned on [ 744.370698] LNetError: 32284:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 744.925033] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 08:08:54 (1786190934) [ 747.347484] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg220-server [ 749.713764] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 08:08:59 (1786190939) [ 749.857789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 749.918527] LNet: 33345:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 749.944392] LNetError: 33345:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 749.979330] LNet: Removed LNI 192.168.202.20@tcp [ 750.025520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 750.160068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 750.202192] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 750.856251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.202.[21-26/2]@tcp [ 750.890435] LNet: 33468:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 750.906680] LNet: 33468:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 750.936363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.202.[21-26/2]@tcp [ 750.942546] LNetError: 33355:0:(lib-lnet.h:1274:lnet_set_route_aliveness()) route to tcp103 through 192.168.202.21@tcp has gone from up to down [ 750.963864] LNetError: 33355:0:(lib-lnet.h:1274:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 758.395372] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 08:09:08 (1786190948) [ 758.516951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 758.556417] LNet: 34053:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 758.561677] LNetError: 34053:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 758.589306] LNet: Removed LNI 192.168.202.20@tcp [ 758.651715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 758.753597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 758.798648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 758.821256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 758.885407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 758.934603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 758.963739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 759.007383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 759.072241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 759.109443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 759.166731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 759.242491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 759.278811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 759.328487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 759.392931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 759.427797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 759.516138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 759.674881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 759.723540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 759.822071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 759.948934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 760.059624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 760.206374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 760.340860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 760.427244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 760.513543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 760.626659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 760.659414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 760.730635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 760.861667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 760.931426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 761.003393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 761.118215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 761.209842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 761.343993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 770.075424] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 08:09:19 (1786190959) [ 772.238516] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg220-server [ 774.041435] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 08:09:24 (1786190964) [ 776.038256] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg220-server [ 778.002540] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 08:09:27 (1786190967) [ 778.305704] LNet: 35459:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 778.322086] LNet: 35459:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 779.265619] Key type .llcrypt unregistered [ 779.271168] Key type ._llcrypt unregistered [ 787.066060] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 787.316411] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 787.320158] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 787.333774] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 790.174934] Key type ._llcrypt registered [ 790.183590] Key type .llcrypt registered [ 790.435399] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 790.451702] alg: No test for adler32 (adler32-zlib) [ 791.461475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 791.527594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 791.564183] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 792.390941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 792.418937] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 794.788758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 794.820498] LNet: Removed LNI 10.1.2.3@tcp [ 794.957511] LNet: 36187:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 794.969285] LNetError: 36187:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 794.988027] LNet: Removed LNI 192.168.202.20@tcp [ 795.614462] Key type .llcrypt unregistered [ 795.621137] Key type ._llcrypt unregistered [ 808.996659] Key type ._llcrypt registered [ 809.005297] Key type .llcrypt registered [ 810.669057] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 08:10:00 (1786191000) [ 811.191579] Key type .llcrypt unregistered [ 811.193323] Key type ._llcrypt unregistered [ 818.087301] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 818.302310] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 818.310990] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 818.321460] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 821.025226] Key type ._llcrypt registered [ 821.028594] Key type .llcrypt registered [ 821.303838] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 821.316367] alg: No test for adler32 (adler32-zlib) [ 822.310856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 822.376379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 822.404403] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 823.173375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 823.194703] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 826.189554] LNet: 37690:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 826.196968] LNetError: 37690:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 826.203981] LNet: Removed LNI 10.1.2.3@tcp [ 826.236930] LNet: Removed LNI 192.168.202.20@tcp [ 827.259651] Key type .llcrypt unregistered [ 827.263664] Key type ._llcrypt unregistered [ 841.363358] Key type ._llcrypt registered [ 841.365531] Key type .llcrypt registered [ 842.841117] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 08:10:32 (1786191032) [ 843.373728] Key type .llcrypt unregistered [ 843.378198] Key type ._llcrypt unregistered [ 850.465906] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 850.623675] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 850.626790] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 850.631087] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 853.261680] Key type ._llcrypt registered [ 853.266861] Key type .llcrypt registered [ 853.447983] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 853.466284] alg: No test for adler32 (adler32-zlib) [ 854.442473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 854.498594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 854.547839] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 856.557575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 856.581785] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 859.143885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 859.167506] LNet: Removed LNI 10.1.2.3@tcp [ 859.195155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 859.244547] LNet: Removed LNI 10.1.2.31@tcp [ 859.265101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 859.302273] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 860.553045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 860.609670] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 861.907244] LNet: 39618:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 861.922148] LNetError: 39618:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 861.931620] LNet: Removed LNI 10.1.2.31@tcp [ 861.949105] LNet: Removed LNI 10.1.2.3@tcp [ 862.464699] Key type .llcrypt unregistered [ 862.466934] Key type ._llcrypt unregistered [ 874.677913] Key type ._llcrypt registered [ 874.681340] Key type .llcrypt registered [ 875.956134] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 08:11:06 (1786191066) [ 876.511956] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 876.528201] alg: No test for adler32 (adler32-zlib) [ 877.466922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 877.510739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 877.531748] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 878.353261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 878.379596] LNet: 40575:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 878.391792] LNetError: 40575:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 878.416986] LNet: Removed LNI 192.168.202.20@tcp [ 878.436881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 878.499721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.202.20@tcp [ 878.519549] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 879.234058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.202.20@tcp [ 879.261863] LNet: Removed LNI 192.168.202.20@tcp [ 879.322655] LNet: 40710:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 879.939517] Key type .llcrypt unregistered [ 879.943958] Key type ._llcrypt unregistered [ 891.695515] Key type ._llcrypt registered [ 891.699882] Key type .llcrypt registered [ 893.017346] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 08:11:23 (1786191083) [ 894.812953] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg220-server [ 896.474913] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 08:11:26 (1786191086) [ 897.148431] Key type .llcrypt unregistered [ 897.154830] Key type ._llcrypt unregistered [ 903.372824] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 903.641589] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 903.647606] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 903.653336] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 906.239292] Key type ._llcrypt registered [ 906.241691] Key type .llcrypt registered [ 906.575574] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 906.618736] alg: No test for adler32 (adler32-zlib) [ 907.622777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 907.703866] LNet: Added LNI 192.168.202.20@tcp1 [8/256/0/180] [ 908.579854] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 910.765711] LNet: 42556:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 910.771365] LNetError: 42556:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 910.776726] LNet: Removed LNI 192.168.202.20@tcp1 [ 910.795236] LNet: Removed LNI 10.1.2.3@tcp2 [ 911.463456] Key type .llcrypt unregistered [ 911.464881] Key type ._llcrypt unregistered [ 922.440891] Key type ._llcrypt registered [ 922.443077] Key type .llcrypt registered [ 923.547942] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 08:11:53 (1786191113) [ 924.232819] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 924.246699] alg: No test for adler32 (adler32-zlib) [ 925.218515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 925.276612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 925.325090] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 926.185478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 926.215400] LNet: 43530:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 926.221532] LNetError: 43530:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 926.249907] LNet: Removed LNI 192.168.202.20@tcp [ 926.268524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 926.336334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 926.358876] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 932.591925] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 08:12:02 (1786191122) [ 932.725262] LNet: 44112:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 932.739363] LNetError: 44112:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 932.756952] LNet: Removed LNI 192.168.202.20@tcp [ 933.406579] Key type .llcrypt unregistered [ 933.408493] Key type ._llcrypt unregistered [ 938.973556] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 939.233955] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 939.238527] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 939.251646] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 941.804627] Key type ._llcrypt registered [ 941.806944] Key type .llcrypt registered [ 942.023534] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 942.040441] alg: No test for adler32 (adler32-zlib) [ 943.014071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 943.090341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 943.143504] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 943.914334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 943.938299] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 945.958714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 945.994307] LNet: 44867:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 946.001979] LNetError: 44867:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 946.008115] LNet: Removed LNI 10.1.2.3@tcp [ 946.019787] LNet: Removed LNI 192.168.202.20@tcp [ 946.040397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 946.099263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 946.122236] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 946.126302] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 946.601264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 946.638459] LNet: 44894:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 946.643594] LNetError: 44894:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 946.648588] LNetError: 44894:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 946.653367] LNet: Removed LNI 10.1.2.3@tcp [ 946.662562] LNet: Removed LNI 192.168.202.20@tcp [ 946.683391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 946.758410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 946.782752] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 946.787841] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 947.128855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 947.146647] LNet: Added LNI 192.168.202.20@tcp2 [8/256/0/180] [ 948.014648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 948.031825] LNet: 45056:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 948.034902] LNetError: 45056:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 948.042625] LNetError: 45056:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 948.047372] LNet: Removed LNI 10.1.2.3@tcp [ 948.049724] LNet: Removed LNI 192.168.202.20@tcp [ 948.059974] LNet: Removed LNI 192.168.202.20@tcp2 [ 948.076555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 948.175692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 948.208614] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 948.217712] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 948.225541] LNet: Added LNI 192.168.202.20@tcp2 [8/256/0/180] [ 955.012623] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 08:12:25 (1786191145) [ 955.128072] LNet: 45646:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 955.142285] LNetError: 45646:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 955.150084] LNetError: 45646:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 955.161081] LNet: Removed LNI 10.1.2.3@tcp [ 955.165523] LNet: Removed LNI 192.168.202.20@tcp [ 955.183261] LNet: Removed LNI 192.168.202.20@tcp2 [ 955.831333] Key type .llcrypt unregistered [ 955.834031] Key type ._llcrypt unregistered [ 962.659460] Key type ._llcrypt registered [ 962.670166] Key type .llcrypt registered [ 962.978223] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 963.005076] alg: No test for adler32 (adler32-zlib) [ 963.932785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 964.010899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 964.034388] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 965.139360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 965.162857] LNet: 46053:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 965.172111] LNetError: 46053:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 965.200435] LNet: Removed LNI 192.168.202.20@tcp [ 965.231647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 965.332113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 965.384515] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 965.965651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 965.980296] LNet: 46083:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 965.986359] LNetError: 46083:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 965.997771] LNet: Removed LNI 192.168.202.20@tcp [ 966.022710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 966.082671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 966.101293] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 966.500256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 966.527321] LNet: Removed LNI 192.168.202.20@tcp [ 966.547973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 966.604357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 966.622922] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 967.175807] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 967.446427] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 967.451752] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 967.455430] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 969.728945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 969.739333] LNet: 46193:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 969.742263] LNet: 46193:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 969.745666] LNetError: 46193:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 969.748460] LNetError: 46193:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 969.758597] LNet: Removed LNI 192.168.202.20@tcp [ 969.787726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 969.853547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 969.899441] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 977.220574] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 08:12:47 (1786191167) [ 977.392504] LNet: 46786:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 977.401713] LNetError: 46786:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 977.418994] LNet: Removed LNI 192.168.202.20@tcp [ 978.069776] Key type .llcrypt unregistered [ 978.072421] Key type ._llcrypt unregistered [ 985.167341] Key type ._llcrypt registered [ 985.177677] Key type .llcrypt registered [ 985.418855] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 985.439609] alg: No test for adler32 (adler32-zlib) [ 986.443102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 986.507841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 986.537980] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 987.421164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 987.466348] LNet: 47193:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 987.472391] LNetError: 47193:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 987.490262] LNet: Removed LNI 192.168.202.20@tcp [ 987.508716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 987.580684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 987.618713] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 994.320669] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 08:13:04 (1786191184) [ 994.834695] LNet: 47785:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 994.845967] LNetError: 47785:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 994.870887] LNet: Removed LNI 192.168.202.20@tcp [ 995.438453] Key type .llcrypt unregistered [ 995.440263] Key type ._llcrypt unregistered [ 1001.083907] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1001.308547] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1001.319210] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1001.328139] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1003.978811] Key type ._llcrypt registered [ 1003.982750] Key type .llcrypt registered [ 1004.247847] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1004.274817] alg: No test for adler32 (adler32-zlib) [ 1005.294535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1005.368274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1005.415779] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 1006.628629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1006.656742] LNet: 48235:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1006.676727] LNetError: 48235:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1006.694319] LNet: Removed LNI 192.168.202.20@tcp [ 1006.754568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1006.837487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 1006.867346] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 1014.208656] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 08:13:24 (1786191204) [ 1014.520052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1014.552106] LNet: 48834:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1014.558360] LNetError: 48834:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1014.599405] LNet: Removed LNI 192.168.202.20@tcp [ 1014.623591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1014.775478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 1020.218594] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 08:13:30 (1786191210) [ 1020.662810] LNet: 49416:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1021.371044] Key type .llcrypt unregistered [ 1021.372575] Key type ._llcrypt unregistered [ 1027.829732] Key type ._llcrypt registered [ 1027.834794] Key type .llcrypt registered [ 1028.157870] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1028.173847] alg: No test for adler32 (adler32-zlib) [ 1029.132689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1029.215422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1029.245933] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 1030.388717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1030.407430] LNet: 49818:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1030.411943] LNetError: 49818:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1030.426071] LNet: Removed LNI 192.168.202.20@tcp [ 1030.439868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1030.499537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 1030.530545] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 1036.259575] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 08:13:46 (1786191226) [ 1036.543636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1036.568863] LNet: 50413:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1036.575239] LNetError: 50413:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1036.600759] LNet: Removed LNI 192.168.202.20@tcp [ 1036.628544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1036.680616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 1041.971511] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 08:13:52 (1786191232) [ 1043.114643] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 1044.610148] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 08:13:54 (1786191234) [ 1045.872138] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 1049.296535] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 08:13:59 (1786191239) [ 1050.538045] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 1052.077297] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 08:14:02 (1786191242) [ 1053.371547] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1054.818956] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 08:14:05 (1786191245) [ 1056.135079] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1057.558965] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 08:14:08 (1786191248) [ 1057.726923] LNet: 52466:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1058.199687] Key type .llcrypt unregistered [ 1058.201478] Key type ._llcrypt unregistered [ 1063.555842] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1063.882906] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1063.889281] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1063.896987] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1066.415759] Key type ._llcrypt registered [ 1066.417338] Key type .llcrypt registered [ 1066.547660] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1066.563826] alg: No test for adler32 (adler32-zlib) [ 1067.493215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1067.569344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1067.590320] LNet: Added LNI 192.168.202.20@tcp [8/512/0/180] [ 1067.595048] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1067.598714] LNet: 52791:0:(api-ni.c:2985:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1067.689680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1067.712705] LNet: 52810:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1067.718948] LNetError: 52810:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1067.723789] LNet: Removed LNI 10.1.2.3@tcp [ 1067.725996] LNetError: 52810:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1067.742341] LNet: Removed LNI 192.168.202.20@tcp [ 1067.755929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1067.837965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1067.874332] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 1067.938535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1067.958470] LNet: 52834:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1067.964646] LNetError: 52834:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1067.985995] LNet: Removed LNI 192.168.202.20@tcp [ 1068.016211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1068.098303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1068.128848] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 1068.137077] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1068.139990] LNet: 52842:0:(api-ni.c:2985:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1068.252223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1068.272227] LNet: 52860:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1068.278909] LNetError: 52860:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1068.286278] LNet: Removed LNI 10.1.2.3@tcp [ 1068.287912] LNetError: 52860:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1068.299973] LNet: Removed LNI 192.168.202.20@tcp [ 1068.329592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1068.399150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1068.421086] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 1068.424950] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1068.426801] LNet: 52868:0:(api-ni.c:2985:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1068.593091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1068.611887] LNet: 52894:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1068.618170] LNetError: 52894:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1068.624280] LNet: Removed LNI 10.1.2.3@tcp [ 1068.630644] LNetError: 52894:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1068.649701] LNet: Removed LNI 192.168.202.20@tcp [ 1068.668226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1068.721321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1068.742117] LNet: Added LNI 192.168.202.20@tcp [8/256/0/182] [ 1068.745054] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1068.825545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1068.843418] LNet: 52922:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1068.850541] LNetError: 52922:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1068.855275] LNet: Removed LNI 10.1.2.3@tcp [ 1068.857417] LNetError: 52922:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1068.871616] LNet: Removed LNI 192.168.202.20@tcp [ 1068.881917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1068.936906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1068.962420] LNet: Added LNI 192.168.202.20@tcp [8/256/0/182] [ 1068.969135] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1068.971632] LNet: 52930:0:(api-ni.c:2985:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1069.030696] LNet: Removed LNI 10.1.2.3@tcp [ 1069.039959] LNet: Removed LNI 192.168.202.20@tcp [ 1069.631839] Key type .llcrypt unregistered [ 1069.636287] Key type ._llcrypt unregistered [ 1079.072912] Key type ._llcrypt registered [ 1079.075511] Key type .llcrypt registered [ 1080.194890] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 08:14:30 (1786191270) [ 1081.575953] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 1083.173389] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 08:14:33 (1786191273) [ 1083.466291] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1083.478761] alg: No test for adler32 (adler32-zlib) [ 1084.364522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1084.403399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1084.418784] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 1084.958461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.202.20 [ 1084.971433] LNet: 54198:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1084.977928] LNet: Removed LNI 192.168.202.20@tcp [ 1089.085656] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 08:14:39 (1786191279) [ 1089.336919] LNet: 54768:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1089.341951] LNetError: 54768:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1089.360922] LNet: Removed LNI 192.168.202.20@tcp [ 1089.813802] Key type .llcrypt unregistered [ 1089.815238] Key type ._llcrypt unregistered [ 1094.869710] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1095.033721] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1097.625771] Key type ._llcrypt registered [ 1097.628497] Key type .llcrypt registered [ 1097.756872] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1097.780547] alg: No test for adler32 (adler32-zlib) [ 1099.116827] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1104.009524] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 08:14:54 (1786191294) [ 1104.279257] LNet: 55685:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1104.285708] LNetError: 55685:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1104.315269] LNet: Removed LNI 10.1.2.3@tcp [ 1104.892629] Key type .llcrypt unregistered [ 1104.894319] Key type ._llcrypt unregistered [ 1109.881982] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1110.131420] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1110.134147] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1110.137267] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1112.788723] Key type ._llcrypt registered [ 1112.790719] Key type .llcrypt registered [ 1112.961667] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1112.977179] alg: No test for adler32 (adler32-zlib) [ 1113.982490] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1119.657118] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 08:15:09 (1786191309) [ 1120.933791] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1122.455335] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 08:15:12 (1786191312) [ 1122.721825] LNet: 56873:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1122.731761] LNetError: 56873:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1122.745411] LNet: Removed LNI 10.1.2.3@tcp [ 1123.277530] Key type .llcrypt unregistered [ 1123.279751] Key type ._llcrypt unregistered [ 1128.789468] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1128.967975] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1128.976482] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1128.986473] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1131.656109] Key type ._llcrypt registered [ 1131.658585] Key type .llcrypt registered [ 1131.785498] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1131.807803] alg: No test for adler32 (adler32-zlib) [ 1132.699867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1132.763759] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1139.257550] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 08:15:29 (1786191329) [ 1139.711360] LNet: 58107:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1139.717254] LNetError: 58107:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1139.725137] LNet: Removed LNI 10.1.2.3@tcp [ 1140.291068] Key type .llcrypt unregistered [ 1140.293784] Key type ._llcrypt unregistered [ 1145.930135] Key type ._llcrypt registered [ 1145.931875] Key type .llcrypt registered [ 1146.127853] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1146.148788] alg: No test for adler32 (adler32-zlib) [ 1147.105646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1147.161078] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 1154.451601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.120@tcp [ 1155.144629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.20@tcp -d 192.168.202.120@tcp -m GET -e local_interrupt -r 1 [ 1155.180826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.20@tcp -d 192.168.202.20@tcp -m GET -e local_interrupt -r 1 [ 1155.216764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.120@tcp [ 1155.239433] LNet: There was an unexpected network error while writing to 192.168.202.120: rc = -22 [ 1155.363358] LNet: 1 local NIs in recovery (showing 1): 192.168.202.20@tcp [ 1155.495899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1155.528540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1155.560809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1155.589783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1155.614955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1155.644775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1155.675252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1155.706673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1155.723265] LNet: 58549:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1155.738670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1155.772212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1155.800940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1155.829566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1155.864611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1155.904473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1155.935630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1156.552615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.20@tcp -d 192.168.202.120@tcp -m GET -e local_dropped -r 1 [ 1156.593541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.20@tcp -d 192.168.202.20@tcp -m GET -e local_dropped -r 1 [ 1156.635972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.120@tcp [ 1156.654275] LNet: There was an unexpected network error while writing to 192.168.202.120: rc = -22 [ 1156.855160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1156.882609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1156.918474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1156.950539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1156.976784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1157.001362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1157.030954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1157.064201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1157.078069] LNet: 58658:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1157.093753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1157.113528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1157.141364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1157.170267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1157.191403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1157.222489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1157.251123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1157.704375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.20@tcp -d 192.168.202.120@tcp -m GET -e local_aborted -r 1 [ 1157.737557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.20@tcp -d 192.168.202.20@tcp -m GET -e local_aborted -r 1 [ 1157.780300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.120@tcp [ 1157.795672] LNet: There was an unexpected network error while writing to 192.168.202.120: rc = -22 [ 1157.984329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1158.015766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1158.036483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1158.061320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1158.083579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1158.112481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1158.134705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1158.154506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1158.169630] LNet: 58767:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1158.181612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1158.203418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1158.222171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1158.245365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1158.278349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1158.317819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1158.343228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1158.788638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.20@tcp -d 192.168.202.120@tcp -m GET -e local_no_route -r 1 [ 1158.814516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.20@tcp -d 192.168.202.20@tcp -m GET -e local_no_route -r 1 [ 1158.846788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.120@tcp [ 1159.061377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1159.083434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1159.107804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1159.137514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1159.182336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1159.210670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1159.241468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1159.268663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1159.292647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1159.321830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1159.367237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1159.388599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1159.409597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1159.438547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1159.460877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1159.864213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.20@tcp -d 192.168.202.120@tcp -m GET -e local_timeout -r 1 [ 1159.890818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.20@tcp -d 192.168.202.20@tcp -m GET -e local_timeout -r 1 [ 1159.918929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.120@tcp [ 1159.940849] LNet: There was an unexpected network error while writing to 192.168.202.120: rc = -22 [ 1159.945492] LNet: Skipped 1 previous similar message [ 1160.114967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1160.142200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1160.161949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1160.179885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1160.196238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1160.226028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1160.268432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1160.299442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1160.319685] LNet: 58986:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1160.326525] LNet: 58986:0:(api-ni.c:295:recovery_interval_set()) Skipped 1 previous similar message [ 1160.348300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1160.382809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1160.411362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1160.447493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1160.474777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1160.516811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1160.553727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1161.021597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.20@tcp -d 192.168.202.120@tcp -m GET -e local_error -r 1 [ 1161.050416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.20@tcp -d 192.168.202.20@tcp -m GET -e local_error -r 1 [ 1161.083266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.120@tcp [ 1161.296776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1161.313619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1161.328350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1161.344578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1161.362979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1161.379742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1161.401186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1161.420992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1161.441706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1161.458802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1161.486698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1161.512705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1161.537962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1161.560970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1161.591795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1161.817758] LNet: 59140:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1161.824166] LNetError: 59140:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1162.866814] LNet: Removed LNI 192.168.202.20@tcp [ 1163.194723] Key type .llcrypt unregistered [ 1163.198378] Key type ._llcrypt unregistered [ 1176.085138] Key type ._llcrypt registered [ 1176.087238] Key type .llcrypt registered [ 1177.038218] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 08:16:07 (1786191367) [ 1177.634431] Key type .llcrypt unregistered [ 1177.636613] Key type ._llcrypt unregistered [ 1182.767967] Key type ._llcrypt registered [ 1182.770279] Key type .llcrypt registered [ 1182.913236] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1182.923942] alg: No test for adler32 (adler32-zlib) [ 1183.808402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1183.827818] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 1190.010244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.120@tcp [ 1198.150409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1198.180908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1198.203937] LNet: Added LNI 192.168.202.20@tcp1 [8/256/0/180] [ 1199.285350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.20@tcp -d 192.168.202.120@tcp -m GET -e local_interrupt -r 1 [ 1199.307877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.20@tcp -d 192.168.202.120@tcp1 -m GET -e local_interrupt -r 1 [ 1199.338434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.20@tcp -d 192.168.202.20@tcp -m GET -e local_interrupt -r 1 [ 1199.364152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.20@tcp -d 192.168.202.20@tcp1 -m GET -e local_interrupt -r 1 [ 1199.394772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.20@tcp1 -d 192.168.202.120@tcp -m GET -e local_interrupt -r 1 [ 1199.422932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.20@tcp1 -d 192.168.202.120@tcp1 -m GET -e local_interrupt -r 1 [ 1199.447510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.20@tcp1 -d 192.168.202.20@tcp -m GET -e local_interrupt -r 1 [ 1199.473361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.20@tcp1 -d 192.168.202.20@tcp1 -m GET -e local_interrupt -r 1 [ 1199.497500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.120@tcp [ 1199.520367] LNet: There was an unexpected network error while writing to 192.168.202.120: rc = -22 [ 1199.525490] LNet: 1 local NIs in recovery (showing 1): 192.168.202.20@tcp [ 1199.717449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1199.739613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1199.771275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1199.792056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1199.813596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1199.832303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1199.857939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1199.876509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1199.897198] LNet: 60609:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1199.908731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1199.943823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1199.972772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1200.000892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1200.040167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1200.060893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1200.081675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1200.430645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.20@tcp -d 192.168.202.120@tcp -m GET -e local_dropped -r 1 [ 1200.456524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.20@tcp -d 192.168.202.120@tcp1 -m GET -e local_dropped -r 1 [ 1200.487420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.20@tcp -d 192.168.202.20@tcp -m GET -e local_dropped -r 1 [ 1200.515241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.20@tcp -d 192.168.202.20@tcp1 -m GET -e local_dropped -r 1 [ 1200.539231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.20@tcp1 -d 192.168.202.120@tcp -m GET -e local_dropped -r 1 [ 1200.563577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.20@tcp1 -d 192.168.202.120@tcp1 -m GET -e local_dropped -r 1 [ 1200.593728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.20@tcp1 -d 192.168.202.20@tcp -m GET -e local_dropped -r 1 [ 1200.622508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.20@tcp1 -d 192.168.202.20@tcp1 -m GET -e local_dropped -r 1 [ 1200.647388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.120@tcp [ 1200.665793] LNet: There was an unexpected network error while writing to 192.168.202.120: rc = -22 [ 1200.669727] LNet: Skipped 2 previous similar messages [ 1200.812926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1200.841890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1200.860552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1200.888860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1200.910933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1200.932796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1200.951170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1200.973573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1200.986030] LNet: 60731:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1200.997123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1201.014188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1201.041589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1201.065606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1201.088398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1201.111860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1201.137417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1201.472302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.20@tcp -d 192.168.202.120@tcp -m GET -e local_aborted -r 1 [ 1201.496873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.20@tcp -d 192.168.202.120@tcp1 -m GET -e local_aborted -r 1 [ 1201.544861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.20@tcp -d 192.168.202.20@tcp -m GET -e local_aborted -r 1 [ 1201.584316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.20@tcp -d 192.168.202.20@tcp1 -m GET -e local_aborted -r 1 [ 1201.650840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.20@tcp1 -d 192.168.202.120@tcp -m GET -e local_aborted -r 1 [ 1201.687882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.20@tcp1 -d 192.168.202.120@tcp1 -m GET -e local_aborted -r 1 [ 1201.730144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.20@tcp1 -d 192.168.202.20@tcp -m GET -e local_aborted -r 1 [ 1201.760891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.20@tcp1 -d 192.168.202.20@tcp1 -m GET -e local_aborted -r 1 [ 1201.801552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.120@tcp [ 1201.826200] LNet: There was an unexpected network error while writing to 192.168.202.120: rc = -22 [ 1201.837085] LNet: Skipped 2 previous similar messages [ 1202.035662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1202.070058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1202.091888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1202.108628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1202.164276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1202.190762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1202.219916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1202.245603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1202.266274] LNet: 60852:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1202.285355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1202.315603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1202.349658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1202.375672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1202.404038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1202.429810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1202.459510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1202.809828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.20@tcp -d 192.168.202.120@tcp -m GET -e local_no_route -r 1 [ 1202.839190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.20@tcp -d 192.168.202.120@tcp1 -m GET -e local_no_route -r 1 [ 1202.873952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.20@tcp -d 192.168.202.20@tcp -m GET -e local_no_route -r 1 [ 1202.902918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.20@tcp -d 192.168.202.20@tcp1 -m GET -e local_no_route -r 1 [ 1202.941074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.20@tcp1 -d 192.168.202.120@tcp -m GET -e local_no_route -r 1 [ 1202.969251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.20@tcp1 -d 192.168.202.120@tcp1 -m GET -e local_no_route -r 1 [ 1203.003869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.20@tcp1 -d 192.168.202.20@tcp -m GET -e local_no_route -r 1 [ 1203.035389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.20@tcp1 -d 192.168.202.20@tcp1 -m GET -e local_no_route -r 1 [ 1203.071479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.120@tcp [ 1203.266215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1203.291206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1203.326918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1203.358349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1203.400381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1203.435293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1203.461894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1203.486396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1203.511919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1203.541745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1203.569368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1203.592738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1203.618143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1203.645841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1203.664775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1203.953723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.20@tcp -d 192.168.202.120@tcp -m GET -e local_timeout -r 1 [ 1203.981482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.20@tcp -d 192.168.202.120@tcp1 -m GET -e local_timeout -r 1 [ 1204.005448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.20@tcp -d 192.168.202.20@tcp -m GET -e local_timeout -r 1 [ 1204.025331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.20@tcp -d 192.168.202.20@tcp1 -m GET -e local_timeout -r 1 [ 1204.051787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.20@tcp1 -d 192.168.202.120@tcp -m GET -e local_timeout -r 1 [ 1204.074264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.20@tcp1 -d 192.168.202.120@tcp1 -m GET -e local_timeout -r 1 [ 1204.092731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.20@tcp1 -d 192.168.202.20@tcp -m GET -e local_timeout -r 1 [ 1204.114286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.20@tcp1 -d 192.168.202.20@tcp1 -m GET -e local_timeout -r 1 [ 1204.147627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.120@tcp [ 1204.161997] LNet: There was an unexpected network error while writing to 192.168.202.120: rc = -22 [ 1204.165492] LNet: Skipped 5 previous similar messages [ 1204.297822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1204.328172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1204.351058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1204.372585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1204.395300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1204.417184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1204.442508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1204.475297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1204.504575] LNet: 61094:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1204.509200] LNet: 61094:0:(api-ni.c:295:recovery_interval_set()) Skipped 1 previous similar message [ 1204.533559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1204.566993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1204.595437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1204.619608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1204.646196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1204.668868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1204.695260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1205.071611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.20@tcp -d 192.168.202.120@tcp -m GET -e local_error -r 1 [ 1205.134147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.20@tcp -d 192.168.202.120@tcp1 -m GET -e local_error -r 1 [ 1205.160795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.20@tcp -d 192.168.202.20@tcp -m GET -e local_error -r 1 [ 1205.188370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.20@tcp -d 192.168.202.20@tcp1 -m GET -e local_error -r 1 [ 1205.214232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.20@tcp1 -d 192.168.202.120@tcp -m GET -e local_error -r 1 [ 1205.242349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.20@tcp1 -d 192.168.202.120@tcp1 -m GET -e local_error -r 1 [ 1205.270674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.20@tcp1 -d 192.168.202.20@tcp -m GET -e local_error -r 1 [ 1205.299995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.20@tcp1 -d 192.168.202.20@tcp1 -m GET -e local_error -r 1 [ 1205.325785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.120@tcp [ 1205.545159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1205.570860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1205.598942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1205.619919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1205.638606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1205.672760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1205.707520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1205.743682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1205.780296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1205.811287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1205.843945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1205.868967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1205.898327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1205.927989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1205.949963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1210.438728] LNet: 61296:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1210.444252] LNetError: 61296:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1210.452781] LNet: Removed LNI 192.168.202.20@tcp [ 1210.469201] LNet: Removed LNI 192.168.202.20@tcp1 [ 1210.702624] Key type .llcrypt unregistered [ 1210.704804] Key type ._llcrypt unregistered [ 1221.504784] Key type ._llcrypt registered [ 1221.507911] Key type .llcrypt registered [ 1222.407180] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 08:16:52 (1786191412) [ 1223.052493] Key type .llcrypt unregistered [ 1223.055069] Key type ._llcrypt unregistered [ 1227.381780] Key type ._llcrypt registered [ 1227.384813] Key type .llcrypt registered [ 1227.550597] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1227.568848] alg: No test for adler32 (adler32-zlib) [ 1228.481641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1228.518512] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 1233.687284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.120@tcp [ 1234.122859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.20@tcp -d 192.168.202.120@tcp -m GET -e remote_dropped -r 1 [ 1234.155670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.20@tcp -d 192.168.202.20@tcp -m GET -e remote_dropped -r 1 [ 1234.180937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.120@tcp [ 1234.193796] LNet: There was an unexpected network error while writing to 192.168.202.120: rc = -22 [ 1234.316055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1234.335866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1234.382369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1234.410855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1234.443782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1234.472410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1234.507181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1234.535814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1234.549772] LNet: 62547:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1234.562433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1234.589940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1234.618584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1234.644509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1234.671552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1234.696867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1234.724723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1235.090996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.20@tcp -d 192.168.202.120@tcp -m GET -e remote_error -r 1 [ 1235.117950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.20@tcp -d 192.168.202.20@tcp -m GET -e remote_error -r 1 [ 1235.150220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.120@tcp [ 1235.166737] LNet: There was an unexpected network error while writing to 192.168.202.120: rc = -22 [ 1235.333416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1235.362269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1235.392931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1235.415635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1235.436424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1235.466706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1235.495766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1235.521435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1235.535033] LNet: 62656:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1235.548900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1235.563248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1235.589474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1235.613946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1235.638957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1235.658810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1235.687503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1236.008900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.20@tcp -d 192.168.202.120@tcp -m GET -e remote_timeout -r 1 [ 1236.040703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.20@tcp -d 192.168.202.20@tcp -m GET -e remote_timeout -r 1 [ 1236.060466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.120@tcp [ 1236.208311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1236.230795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1236.253617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1236.274337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1236.295796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1236.323151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1236.346357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1236.366588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1236.389590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1236.408829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1236.431980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1236.456897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1236.478810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1236.501858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1236.524417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1236.689877] LNet: 62811:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1236.695375] LNetError: 62811:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1237.742995] LNet: Removed LNI 192.168.202.20@tcp [ 1238.019680] Key type .llcrypt unregistered [ 1238.022888] Key type ._llcrypt unregistered [ 1251.464972] Key type ._llcrypt registered [ 1251.466853] Key type .llcrypt registered [ 1252.487360] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 08:17:22 (1786191442) [ 1253.168122] Key type .llcrypt unregistered [ 1253.171527] Key type ._llcrypt unregistered [ 1258.254618] Key type ._llcrypt registered [ 1258.257302] Key type .llcrypt registered [ 1258.402970] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1258.415692] alg: No test for adler32 (adler32-zlib) [ 1259.293075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1259.314569] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 1265.325594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.120@tcp [ 1271.856935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1271.887450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1271.904368] LNet: Added LNI 192.168.202.20@tcp1 [8/256/0/180] [ 1272.967363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.20@tcp -d 192.168.202.120@tcp -m GET -e remote_dropped -r 1 [ 1273.025320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.20@tcp -d 192.168.202.120@tcp1 -m GET -e remote_dropped -r 1 [ 1273.079971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.20@tcp -d 192.168.202.20@tcp -m GET -e remote_dropped -r 1 [ 1273.118772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.20@tcp -d 192.168.202.20@tcp1 -m GET -e remote_dropped -r 1 [ 1273.146389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.20@tcp1 -d 192.168.202.120@tcp -m GET -e remote_dropped -r 1 [ 1273.178213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.20@tcp1 -d 192.168.202.120@tcp1 -m GET -e remote_dropped -r 1 [ 1273.204144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.20@tcp1 -d 192.168.202.20@tcp -m GET -e remote_dropped -r 1 [ 1273.232319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.20@tcp1 -d 192.168.202.20@tcp1 -m GET -e remote_dropped -r 1 [ 1273.267520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.120@tcp [ 1273.283517] LNet: There was an unexpected network error while writing to 192.168.202.120: rc = -22 [ 1273.289814] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.120@tcp [ 1273.507719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1273.548932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1273.579803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1273.621438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1273.658982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1273.683622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1273.710107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1273.738771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1273.751731] LNet: 64280:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1273.768811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1273.796559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1273.821228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1273.845944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1273.872545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1273.899244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1273.927802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1273.965752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1274.354720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.20@tcp -d 192.168.202.120@tcp -m GET -e remote_error -r 1 [ 1274.385170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.20@tcp -d 192.168.202.120@tcp1 -m GET -e remote_error -r 1 [ 1274.404731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.20@tcp -d 192.168.202.20@tcp -m GET -e remote_error -r 1 [ 1274.436645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.20@tcp -d 192.168.202.20@tcp1 -m GET -e remote_error -r 1 [ 1274.465610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.20@tcp1 -d 192.168.202.120@tcp -m GET -e remote_error -r 1 [ 1274.493121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.20@tcp1 -d 192.168.202.120@tcp1 -m GET -e remote_error -r 1 [ 1274.513214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.20@tcp1 -d 192.168.202.20@tcp -m GET -e remote_error -r 1 [ 1274.548884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.20@tcp1 -d 192.168.202.20@tcp1 -m GET -e remote_error -r 1 [ 1274.579418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.120@tcp [ 1274.600631] LNet: There was an unexpected network error while writing to 192.168.202.120: rc = -22 [ 1274.605550] LNet: Skipped 2 previous similar messages [ 1274.747971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1274.772515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1274.790262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1274.809645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1274.830488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1274.848957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1274.866953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1274.887399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1274.897964] LNet: 64403:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1274.907814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1274.923242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1274.939800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1274.958128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1274.974971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1274.993796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1275.011842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1275.032741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1275.370626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.20@tcp -d 192.168.202.120@tcp -m GET -e remote_timeout -r 1 [ 1275.405078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.20@tcp -d 192.168.202.120@tcp1 -m GET -e remote_timeout -r 1 [ 1275.450506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.20@tcp -d 192.168.202.20@tcp -m GET -e remote_timeout -r 1 [ 1275.483779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.20@tcp -d 192.168.202.20@tcp1 -m GET -e remote_timeout -r 1 [ 1275.517287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.20@tcp1 -d 192.168.202.120@tcp -m GET -e remote_timeout -r 1 [ 1275.546792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.20@tcp1 -d 192.168.202.120@tcp1 -m GET -e remote_timeout -r 1 [ 1275.576118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.20@tcp1 -d 192.168.202.20@tcp -m GET -e remote_timeout -r 1 [ 1275.601870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.20@tcp1 -d 192.168.202.20@tcp1 -m GET -e remote_timeout -r 1 [ 1275.631396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.120@tcp [ 1275.641528] LNet: There was an unexpected network error while writing to 192.168.202.120: rc = -22 [ 1275.762840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1275.780515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1275.798494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1275.819738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1275.845557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1275.866424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1275.889684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1275.911717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1275.928351] LNet: 64526:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1275.945047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1275.959862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1275.974404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1275.994321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1276.009415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1276.029569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1276.056377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1276.077608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1279.753575] LNet: 64607:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1279.758989] LNetError: 64607:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1280.799496] LNet: Removed LNI 192.168.202.20@tcp [ 1280.804457] LNetError: 64607:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1280.831939] LNet: Removed LNI 192.168.202.20@tcp1 [ 1281.237488] Key type .llcrypt unregistered [ 1281.238900] Key type ._llcrypt unregistered [ 1294.076771] Key type ._llcrypt registered [ 1294.078080] Key type .llcrypt registered [ 1295.036748] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 08:18:05 (1786191485) [ 1296.178452] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1297.460827] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 08:18:08 (1786191488) [ 1298.150248] Key type .llcrypt unregistered [ 1298.152585] Key type ._llcrypt unregistered [ 1302.771562] Key type ._llcrypt registered [ 1302.773524] Key type .llcrypt registered [ 1302.918323] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1302.928772] alg: No test for adler32 (adler32-zlib) [ 1303.832393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1303.853614] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 1309.467727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.120@tcp [ 1309.893671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.20@tcp -d 192.168.202.120@tcp -m GET -e network_timeout -r 1 [ 1309.913816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.20@tcp -d 192.168.202.20@tcp -m GET -e network_timeout -r 1 [ 1309.933522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.120@tcp [ 1309.942863] LNet: There was an unexpected network error while writing to 192.168.202.120: rc = -22 [ 1309.983680] LNet: 1 local NIs in recovery (showing 1): 192.168.202.20@tcp [ 1310.081119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1310.104971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1310.127952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1310.152228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1310.174286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1310.193201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1310.214264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1310.263421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1310.277176] LNet: 66153:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1310.287359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1310.313989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1310.334935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1310.356671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1310.391250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1310.424534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1310.617131] LNet: 66196:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1310.621917] LNetError: 66196:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1310.638242] LNet: Removed LNI 192.168.202.20@tcp [ 1310.926451] Key type .llcrypt unregistered [ 1310.928831] Key type ._llcrypt unregistered [ 1322.075598] Key type ._llcrypt registered [ 1322.077125] Key type .llcrypt registered [ 1322.177642] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1322.187556] alg: No test for adler32 (adler32-zlib) [ 1323.038029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1323.070512] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 1328.371337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.120@tcp [ 1333.908687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1333.931477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1333.950799] LNet: Added LNI 192.168.202.20@tcp1 [8/256/0/180] [ 1335.026515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.20@tcp -d 192.168.202.120@tcp -m GET -e network_timeout -r 1 [ 1335.055743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.20@tcp -d 192.168.202.120@tcp1 -m GET -e network_timeout -r 1 [ 1335.089257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.20@tcp -d 192.168.202.20@tcp -m GET -e network_timeout -r 1 [ 1335.111459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.20@tcp -d 192.168.202.20@tcp1 -m GET -e network_timeout -r 1 [ 1335.133526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.20@tcp1 -d 192.168.202.120@tcp -m GET -e network_timeout -r 1 [ 1335.156087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.20@tcp1 -d 192.168.202.120@tcp1 -m GET -e network_timeout -r 1 [ 1335.178799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.20@tcp1 -d 192.168.202.20@tcp -m GET -e network_timeout -r 1 [ 1335.205743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.20@tcp1 -d 192.168.202.20@tcp1 -m GET -e network_timeout -r 1 [ 1335.235703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.120@tcp [ 1335.248904] LNet: There was an unexpected network error while writing to 192.168.202.120: rc = -22 [ 1335.392276] LNet: 1 local NIs in recovery (showing 1): 192.168.202.20@tcp [ 1335.395425] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.120@tcp [ 1335.402966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1335.423497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1335.443461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1335.464522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1335.482498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1335.500599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1335.518899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1335.536609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1335.547317] LNet: 67077:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1335.559567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1335.584390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1335.604318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1335.637885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1335.665073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1335.700859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1338.602756] LNet: There was an unexpected network error while writing to 192.168.202.120: rc = -22 [ 1338.752403] LNet: 67155:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1338.757514] LNetError: 67155:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1338.763671] LNet: Removed LNI 192.168.202.20@tcp [ 1338.777496] LNet: Removed LNI 192.168.202.20@tcp1 [ 1338.995724] Key type .llcrypt unregistered [ 1338.998985] Key type ._llcrypt unregistered [ 1348.081585] Key type ._llcrypt registered [ 1348.083107] Key type .llcrypt registered [ 1348.670946] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 08:18:59 (1786191539) [ 1349.031981] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1349.040995] alg: No test for adler32 (adler32-zlib) [ 1349.864307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1349.898193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1349.912870] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 1350.335112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1350.343797] LNet: Added LNI 192.168.202.20@tcp1 [8/256/0/180] [ 1350.834952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.20@tcp [ 1350.880217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1350.907694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1350.936800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1350.964794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1350.995340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1351.019270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.20@tcp [ 1351.036199] LNetError: 68253:0:(api-ni.c:4472:lnet_ni_set_healthv()) manually adding local NI 192.168.202.20@tcp to recovery [ 1351.903644] LNet: 1 local NIs in recovery (showing 1): 192.168.202.20@tcp [ 1351.904739] LNet: There was an unexpected network error while writing to 192.168.202.20: rc = -22 [ 1353.951560] LNet: There was an unexpected network error while writing to 192.168.202.20: rc = -22 [ 1354.376267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1357.023354] LNet: There was an unexpected network error while writing to 192.168.202.20: rc = -22 [ 1357.680474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1357.701964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1357.729739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1357.744605] LNet: 68360:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1357.749213] LNetError: 68360:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1357.756202] LNet: Removed LNI 192.168.202.20@tcp [ 1357.777221] LNet: Removed LNI 192.168.202.20@tcp1 [ 1357.805743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1357.862944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1357.885317] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 1358.355179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1358.366411] LNet: Added LNI 192.168.202.20@tcp1 [8/256/0/180] [ 1359.032628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.20@tcp [ 1359.089871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1359.129625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1359.183264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1359.214807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1359.246587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1359.274349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1359.315565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.20@tcp [ 1359.330710] LNetError: 68632:0:(api-ni.c:4472:lnet_ni_set_healthv()) manually adding local NI 192.168.202.20@tcp to recovery [ 1359.906243] LNet: There was an unexpected network error while writing to 192.168.202.20: rc = -22 [ 1362.701981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1365.026455] LNet: There was an unexpected network error while writing to 192.168.202.20: rc = -22 [ 1365.032286] LNet: Skipped 1 previous similar message [ 1369.230872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1369.252428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1369.272604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1373.109181] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 08:19:23 (1786191563) [ 1373.250052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1373.285688] LNet: 69339:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1373.289610] LNetError: 69339:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1373.292219] LNetError: 69339:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1373.295837] LNet: Removed LNI 192.168.202.20@tcp [ 1373.306697] LNet: Removed LNI 192.168.202.20@tcp1 [ 1373.326234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1373.375505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1373.395186] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 1373.987562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1374.009849] LNet: Added LNI 192.168.202.20@tcp1 [8/256/0/180] [ 1374.772797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.20@tcp [ 1374.811710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1374.834829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1374.862636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1374.889346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1374.920756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1374.946439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.20@tcp --health 0 [ 1375.008359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1375.393976] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.20@tcp [ 1375.396660] LNet: There was an unexpected network error while writing to 192.168.202.20: rc = -22 [ 1375.411385] LNet: Skipped 1 previous similar message [ 1389.786911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1389.885119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1389.907783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1389.928544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.20@tcp --health 500 [ 1389.982728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1396.485907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1396.541096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1396.560170] LNet: 69831:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1396.564529] LNetError: 69831:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1396.567932] LNetError: 69831:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1396.573431] LNet: Removed LNI 192.168.202.20@tcp [ 1396.584858] LNet: Removed LNI 192.168.202.20@tcp1 [ 1396.600662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1396.643093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1396.661854] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 1397.187104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1397.207575] LNet: Added LNI 192.168.202.20@tcp1 [8/256/0/180] [ 1398.092723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.20@tcp [ 1398.130224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1398.223602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1398.272435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1398.307908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1398.352761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1398.400474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1398.444792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.20@tcp --health 0 [ 1399.711491] LNet: There was an unexpected network error while writing to 192.168.202.20: rc = -22 [ 1399.717701] LNet: Skipped 3 previous similar messages [ 1400.778833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1410.614366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1410.637761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1410.660656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1416.155779] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 08:20:06 (1786191606) [ 1416.570650] LNet: 70888:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1416.581148] LNetError: 70888:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1416.588138] LNetError: 70888:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1416.598891] LNet: Removed LNI 192.168.202.20@tcp [ 1416.620732] LNet: Removed LNI 192.168.202.20@tcp1 [ 1417.146444] Key type .llcrypt unregistered [ 1417.152499] Key type ._llcrypt unregistered [ 1421.844786] Key type ._llcrypt registered [ 1421.848719] Key type .llcrypt registered [ 1422.077293] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1422.090315] alg: No test for adler32 (adler32-zlib) [ 1422.933851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1422.949661] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 1429.091552] Lustre: DEBUG MARKER: Initial discovery [ 1429.221156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.120@tcp [ 1430.550844] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1430.726519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1430.758386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.120@tcp [ 1430.781803] LNet: There was an unexpected network error while writing to 192.168.202.120: rc = -22 [ 1430.791606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1430.808910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.202.120@tcp -r 1 -m GET -l 3 [ 1430.826992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.202.120@tcp -r 1 -m GET -e local_error [ 1430.847826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.202.120@tcp -r 1 -m PUT -l 6 [ 1431.137069] LNet: 1 local NIs in recovery (showing 1): 192.168.202.20@tcp [ 1431.708231] Lustre: DEBUG MARKER: Force oleg220-client.virtnet to discover 192.168.202.120@tcp (in background) [ 1431.883114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.120@tcp [ 1432.869635] Lustre: DEBUG MARKER: Force oleg220-server to discover 192.168.202.20@tcp [ 1434.706486] Lustre: DEBUG MARKER: Wait for 71655 [ 1434.975456] LNet: There was an unexpected network error while writing to 192.168.202.120: rc = -22 [ 1435.934849] Lustre: DEBUG MARKER: Finished wait on 71655 [ 1435.974977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1436.007099] LNet: There was an unexpected network error while writing to 192.168.202.120: rc = -22 [ 1436.027768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1436.249302] LNet: 72108:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1436.253682] LNetError: 72108:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1436.269731] LNet: Removed LNI 192.168.202.20@tcp [ 1436.549654] Key type .llcrypt unregistered [ 1436.551228] Key type ._llcrypt unregistered [ 1445.777863] Key type ._llcrypt registered [ 1445.778786] Key type .llcrypt registered [ 1446.418133] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 08:20:37 (1786191637) [ 1446.722283] Key type .llcrypt unregistered [ 1446.724558] Key type ._llcrypt unregistered [ 1449.650221] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1449.737414] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1449.740163] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1449.744151] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1452.111956] Key type ._llcrypt registered [ 1452.113638] Key type .llcrypt registered [ 1452.232448] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1452.243799] alg: No test for adler32 (adler32-zlib) [ 1453.077093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1453.116835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1453.130398] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 1453.553113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1453.570688] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1458.394556] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 08:20:49 (1786191649) [ 1458.516513] LNet: 74186:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1458.519823] LNetError: 74186:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1458.523201] LNet: Removed LNI 10.1.2.3@tcp [ 1458.528908] LNet: Removed LNI 192.168.202.20@tcp [ 1458.894598] Key type .llcrypt unregistered [ 1458.898100] Key type ._llcrypt unregistered [ 1462.214868] Key type ._llcrypt registered [ 1462.216803] Key type .llcrypt registered [ 1462.320433] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1462.328590] alg: No test for adler32 (adler32-zlib) [ 1463.158219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1463.180984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1463.194477] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 1463.655823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1463.681518] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1464.405775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.202.20@tcp [ 1465.440225] LNetError: Unexpected error -22 connecting to 192.168.202.20@tcp at host 192.168.202.20:988 [ 1468.695242] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 08:20:59 (1786191659) [ 1469.022763] LNet: 75321:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1469.027755] LNetError: 75321:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1469.034051] LNet: Removed LNI 10.1.2.3@tcp [ 1469.047171] LNet: Removed LNI 192.168.202.20@tcp [ 1469.486635] Key type .llcrypt unregistered [ 1469.488639] Key type ._llcrypt unregistered [ 1473.357885] Key type ._llcrypt registered [ 1473.360237] Key type .llcrypt registered [ 1473.475949] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1473.484594] alg: No test for adler32 (adler32-zlib) [ 1474.372905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1474.415114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1474.427539] LNet: Added LNI 192.168.202.20@tcp1 [8/256/0/180] [ 1475.015532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1475.040581] LNet: Added LNI 192.168.202.20@tcp2 [8/256/0/180] [ 1475.631677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.202.20@tcp1 --nid 192.168.202.20@tcp2 [ 1478.320188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.20@tcp1 192.168.202.20@tcp2 [ 1478.339487] LNetError: 76159:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.202.20@tcp1 to 192.168.202.20@tcp2 - Route table may be misconfigured [ 1478.368904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.20@tcp2 192.168.202.20@tcp1 [ 1482.606362] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 08:21:13 (1786191673) [ 1482.843387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1482.870160] LNet: 76732:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1482.875161] LNetError: 76732:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1482.878283] LNet: Removed LNI 192.168.202.20@tcp1 [ 1483.944464] LNet: Removed LNI 192.168.202.20@tcp2 [ 1483.960719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1483.987102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1483.998023] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 1484.346460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1484.356957] LNet: Added LNI 192.168.202.20@tcp1 [8/256/0/180] [ 1484.893575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.20@tcp [ 1484.929099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.20@tcp 192.168.202.20@tcp [ 1484.952704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.20@tcp1 192.168.202.20@tcp1 [ 1484.981554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.20@tcp -d 192.168.202.20@tcp -e network_timeout [ 1485.007180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.20@tcp -d 192.168.202.20@tcp1 -e network_timeout [ 1485.031213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.20@tcp1 -d 192.168.202.20@tcp -e network_timeout [ 1485.053785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.20@tcp1 -d 192.168.202.20@tcp1 -e network_timeout [ 1485.076507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.20@tcp [ 1485.092536] LNet: There was an unexpected network error while writing to 192.168.202.20: rc = -22 [ 1485.104800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1485.164829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1485.224920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1486.047763] LNet: 1 local NIs in recovery (showing 1): 192.168.202.20@tcp [ 1489.541882] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 08:21:20 (1786191680) [ 1489.677707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1489.686655] LNet: 77593:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1489.689729] LNetError: 77593:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1489.692311] LNetError: 77593:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1489.695487] LNet: Removed LNI 192.168.202.20@tcp [ 1490.728920] LNet: Removed LNI 192.168.202.20@tcp1 [ 1490.752709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1490.807582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1490.990837] LNet: 77634:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1491.215937] Key type .llcrypt unregistered [ 1491.223038] Key type ._llcrypt unregistered [ 1497.459464] Key type ._llcrypt registered [ 1497.461888] Key type .llcrypt registered [ 1498.267583] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 08:21:28 (1786191688) [ 1498.541269] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1498.554518] alg: No test for adler32 (adler32-zlib) [ 1499.408275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1500.128845] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1501.053549] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 08:21:31 (1786191691) [ 1501.216707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1501.230838] LNet: 78730:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1501.246123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1501.279085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1501.294875] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 1501.792921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1501.811850] LNet: Added LNI 192.168.202.20@tcp1 [8/256/0/180] [ 1502.397947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.20@tcp [ 1502.425275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.20@tcp1 [ 1502.444236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.20@tcp1 [ 1506.122643] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 08:21:36 (1786191696) [ 1507.237965] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg220-server [ 1508.154517] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 08:21:38 (1786191698) [ 1509.245835] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg220-server [ 1510.267791] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 08:21:40 (1786191700) [ 1511.408364] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg220-server [ 1512.397147] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 08:21:43 (1786191703) [ 1513.607541] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg220-server [ 1514.762947] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 08:21:45 (1786191705) [ 1515.935303] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg220-server [ 1517.095929] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 08:21:47 (1786191707) [ 1518.398498] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg220-server [ 1519.634719] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 08:21:50 (1786191710) [ 1521.109519] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg220-server [ 1522.466720] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 08:21:52 (1786191712) [ 1524.111613] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg220-server [ 1525.613579] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 08:21:55 (1786191715) [ 1526.811675] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg220-server [ 1528.037816] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 08:21:58 (1786191718) [ 1528.161209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1528.184466] LNet: 82604:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1528.205954] LNetError: 82604:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1528.214888] LNet: Removed LNI 192.168.202.20@tcp [ 1529.255832] LNet: Removed LNI 192.168.202.20@tcp1 [ 1529.274355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1529.299119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1529.315440] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 1529.879260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1529.946605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.20@tcp [ 1530.078464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1530.091745] LNet: 82753:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1530.096234] LNetError: 82753:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1530.104430] LNetError: 82753:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1530.113995] LNet: Removed LNI 192.168.202.20@tcp [ 1530.134479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1530.171777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1530.197712] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 1530.750199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.20@tcp [ 1530.870937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1531.941424] LNet: Removed LNI 192.168.202.20@tcp [ 1531.964461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1532.047661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1532.086224] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 1532.613604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1532.676279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.20@tcp [ 1532.776829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1532.785354] LNet: 83046:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1532.787737] LNet: 83046:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1532.792236] LNetError: 83046:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1532.795572] LNetError: 83046:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1532.809105] LNet: Removed LNI 192.168.202.20@tcp [ 1532.824279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1532.861905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1532.892316] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 1533.373699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.20@tcp [ 1533.496034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1534.566265] LNet: Removed LNI 192.168.202.20@tcp [ 1534.588907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1534.643471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1534.658904] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 1535.126433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1535.168963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.20@tcp [ 1535.259475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1535.272960] LNetError: 83338:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1535.274795] LNetError: 83338:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1535.287627] LNet: Removed LNI 192.168.202.20@tcp [ 1535.303472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1535.338479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1535.352323] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 1535.781710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.20@tcp [ 1535.925406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1536.998655] LNet: Removed LNI 192.168.202.20@tcp [ 1537.022479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1537.066618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1537.089249] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 1537.562247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1537.621705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.20@tcp [ 1537.742261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1537.755615] LNet: 83630:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1537.759859] LNet: 83630:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1538.801273] LNet: Removed LNI 192.168.202.20@tcp [ 1538.812799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1538.875585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1538.915441] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 1539.525367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.20@tcp [ 1539.696159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1539.714832] LNetError: 83776:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1539.718520] LNetError: 83776:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1539.739167] LNet: Removed LNI 192.168.202.20@tcp [ 1539.765449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1539.814398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1539.834861] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 1540.270722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1540.315707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.20@tcp [ 1540.450327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1541.490661] LNet: Removed LNI 192.168.202.20@tcp [ 1541.517880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1541.574972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1541.590479] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 1542.065025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.20@tcp [ 1543.338142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1544.426607] LNet: Removed LNI 192.168.202.20@tcp [ 1544.444939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1544.496874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1544.515352] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 1544.957627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1545.004644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.20@tcp [ 1545.164192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1546.221767] LNet: Removed LNI 192.168.202.20@tcp [ 1546.246764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1546.310498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1546.325665] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 1546.774232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.20@tcp [ 1549.247948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1549.257252] LNet: 84370:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1549.260408] LNet: 84370:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1549.263753] LNetError: 84370:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1549.269983] LNetError: 84370:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1549.284923] LNet: Removed LNI 192.168.202.20@tcp [ 1549.302968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1549.337695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1549.350970] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 1549.806496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1549.849203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1549.871748] LNet: Removed LNI 192.168.202.20@tcp [ 1549.891798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1549.929138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1549.948220] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 1550.425635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1550.481082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1550.501602] LNet: Removed LNI 192.168.202.20@tcp [ 1550.519755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1550.558275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1550.617866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1550.648318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1550.687217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1550.746862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1550.772992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1550.852608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1550.907915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1550.933890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1550.972457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1550.993802] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 1551.469782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1551.490074] LNet: Removed LNI 192.168.202.20@tcp [ 1551.504435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1551.545734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1555.987201] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 08:22:26 (1786191746) [ 1556.148894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1556.182057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1556.233858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1556.251328] LNet: Added LNI 192.168.202.20@tcp231 [8/256/0/180] [ 1557.005152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1557.035881] LNet: Removed LNI 192.168.202.20@tcp231 [ 1557.052825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1557.118186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1557.145661] LNet: Added LNI 192.168.202.20@tcp231 [8/256/0/180] [ 1557.149621] LNet: 85536:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1557.410546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1557.438400] LNet: Removed LNI 192.168.202.20@tcp231 [ 1557.446520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1557.461815] LNet: Added LNI 192.168.202.20@tcp231 [8/256/0/180] [ 1558.229394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1558.247146] LNet: Removed LNI 192.168.202.20@tcp231 [ 1558.262771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1558.304912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1558.318549] LNet: Added LNI 192.168.202.20@tcp231 [8/256/0/0] [ 1558.321347] LNet: 85688:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1558.551474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1558.578924] LNet: Removed LNI 192.168.202.20@tcp231 [ 1558.593622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1558.610702] LNet: Added LNI 192.168.202.20@tcp231 [8/256/0/0] [ 1559.399181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1559.430385] LNet: Removed LNI 192.168.202.20@tcp231 [ 1559.447343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1559.493804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1559.508568] LNet: Added LNI 192.168.202.20@tcp231 [8/256/0/60] [ 1559.512093] LNet: 85840:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1559.706301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1559.725688] LNet: Removed LNI 192.168.202.20@tcp231 [ 1559.738492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1559.750197] LNet: Added LNI 192.168.202.20@tcp231 [8/256/0/60] [ 1560.347745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1560.367764] LNet: Removed LNI 192.168.202.20@tcp231 [ 1560.385588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1560.418358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1560.433041] LNet: Added LNI 192.168.202.20@tcp231 [8/256/0/180] [ 1560.608246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1560.625664] LNet: Removed LNI 192.168.202.20@tcp231 [ 1560.633699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1560.647695] LNet: Added LNI 192.168.202.20@tcp231 [8/256/0/180] [ 1564.090905] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 08:22:34 (1786191754) [ 1564.241056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1564.262607] LNet: Removed LNI 192.168.202.20@tcp231 [ 1564.273910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1564.310498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1564.325846] LNet: Added LNI 192.168.202.20@tcp232 [8/256/0/180] [ 1564.681522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1564.702360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1564.724544] LNet: Removed LNI 192.168.202.20@tcp232 [ 1564.734549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1564.746666] LNet: Added LNI 192.168.202.20@tcp232 [8/256/0/180] [ 1565.083516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1565.112936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1565.131160] LNet: Removed LNI 192.168.202.20@tcp232 [ 1565.138805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1565.154197] LNet: Added LNI 192.168.202.20@tcp232 [8/256/0/180] [ 1565.459615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1565.481861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1565.501570] LNet: Removed LNI 192.168.202.20@tcp232 [ 1565.512826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1565.523287] LNet: Added LNI 192.168.202.20@tcp232 [8/256/0/180] [ 1565.912982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1565.937268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1565.966918] LNet: Removed LNI 192.168.202.20@tcp232 [ 1565.979365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1569.238143] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 08:22:40 (1786191760) [ 1569.519218] LNet: 87827:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1569.524375] LNet: 87827:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 12 previous similar messages [ 1569.918541] Key type .llcrypt unregistered [ 1569.921747] Key type ._llcrypt unregistered [ 1573.399647] Key type ._llcrypt registered [ 1573.401378] Key type .llcrypt registered [ 1573.507458] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1573.516542] alg: No test for adler32 (adler32-zlib) [ 1574.351154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1574.369371] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 1579.014879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.120@tcp [ 1584.076218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1584.107628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1584.121505] LNet: Added LNI 192.168.202.20@tcp1 [8/256/0/180] [ 1584.976538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.20@tcp -d 192.168.202.120@tcp -m GET -e local_interrupt -r 2 [ 1584.995713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.20@tcp -d 192.168.202.120@tcp1 -m GET -e local_interrupt -r 2 [ 1585.015541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.20@tcp -d 192.168.202.20@tcp -m GET -e local_interrupt -r 2 [ 1585.039866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.20@tcp -d 192.168.202.20@tcp1 -m GET -e local_interrupt -r 2 [ 1585.061580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.20@tcp1 -d 192.168.202.120@tcp -m GET -e local_interrupt -r 2 [ 1585.088856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.20@tcp1 -d 192.168.202.120@tcp1 -m GET -e local_interrupt -r 2 [ 1585.127810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.20@tcp1 -d 192.168.202.20@tcp -m GET -e local_interrupt -r 2 [ 1585.158773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.20@tcp1 -d 192.168.202.20@tcp1 -m GET -e local_interrupt -r 2 [ 1585.203422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.20@tcp 192.168.202.120@tcp [ 1585.215817] LNet: There was an unexpected network error while writing to 192.168.202.120: rc = -22 [ 1585.220699] LNet: 1 local NIs in recovery (showing 1): 192.168.202.20@tcp [ 1585.383708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1585.410181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1585.428543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1585.463431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1585.488752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1585.518181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1585.537247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1585.555538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1585.567786] LNet: 88492:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1585.583978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1585.612478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1585.646736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1585.670760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1585.696173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1585.720747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1586.061884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.20@tcp -d 192.168.202.120@tcp -m GET -e local_dropped -r 2 [ 1586.099596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.20@tcp -d 192.168.202.120@tcp1 -m GET -e local_dropped -r 2 [ 1586.132306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.20@tcp -d 192.168.202.20@tcp -m GET -e local_dropped -r 2 [ 1586.165851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.20@tcp -d 192.168.202.20@tcp1 -m GET -e local_dropped -r 2 [ 1586.193048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.20@tcp1 -d 192.168.202.120@tcp -m GET -e local_dropped -r 2 [ 1586.218178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.20@tcp1 -d 192.168.202.120@tcp1 -m GET -e local_dropped -r 2 [ 1586.241430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.20@tcp1 -d 192.168.202.20@tcp -m GET -e local_dropped -r 2 [ 1586.261096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.20@tcp1 -d 192.168.202.20@tcp1 -m GET -e local_dropped -r 2 [ 1586.297792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.20@tcp 192.168.202.120@tcp [ 1586.312611] LNet: There was an unexpected network error while writing to 192.168.202.120: rc = -22 [ 1586.494484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1586.517578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1586.545392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1586.570765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1586.589874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1586.612948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1586.636182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1586.650863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1586.661169] LNet: 88612:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1586.676873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1586.700876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1586.723434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1586.741455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1586.782624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1586.818024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1587.163992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.20@tcp -d 192.168.202.120@tcp -m GET -e local_aborted -r 2 [ 1587.183676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.20@tcp -d 192.168.202.120@tcp1 -m GET -e local_aborted -r 2 [ 1587.217058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.20@tcp -d 192.168.202.20@tcp -m GET -e local_aborted -r 2 [ 1587.246420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.20@tcp -d 192.168.202.20@tcp1 -m GET -e local_aborted -r 2 [ 1587.266110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.20@tcp1 -d 192.168.202.120@tcp -m GET -e local_aborted -r 2 [ 1587.284655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.20@tcp1 -d 192.168.202.120@tcp1 -m GET -e local_aborted -r 2 [ 1587.317672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.20@tcp1 -d 192.168.202.20@tcp -m GET -e local_aborted -r 2 [ 1587.355837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.20@tcp1 -d 192.168.202.20@tcp1 -m GET -e local_aborted -r 2 [ 1587.375418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.20@tcp 192.168.202.120@tcp [ 1587.388485] LNet: There was an unexpected network error while writing to 192.168.202.120: rc = -22 [ 1587.586281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1587.602935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1587.621751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1587.644464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1587.677569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1587.697925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1587.722260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1587.746732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1587.761950] LNet: 88732:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1587.778484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1587.804837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1587.828936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1587.845206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1587.863597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1587.887330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1588.179194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.20@tcp -d 192.168.202.120@tcp -m GET -e local_no_route -r 2 [ 1588.205580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.20@tcp -d 192.168.202.120@tcp1 -m GET -e local_no_route -r 2 [ 1588.233542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.20@tcp -d 192.168.202.20@tcp -m GET -e local_no_route -r 2 [ 1588.252771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.20@tcp -d 192.168.202.20@tcp1 -m GET -e local_no_route -r 2 [ 1588.277767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.20@tcp1 -d 192.168.202.120@tcp -m GET -e local_no_route -r 2 [ 1588.299167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.20@tcp1 -d 192.168.202.120@tcp1 -m GET -e local_no_route -r 2 [ 1588.321569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.20@tcp1 -d 192.168.202.20@tcp -m GET -e local_no_route -r 2 [ 1588.349152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.20@tcp1 -d 192.168.202.20@tcp1 -m GET -e local_no_route -r 2 [ 1588.374555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.20@tcp 192.168.202.120@tcp [ 1588.549846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1588.575333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1588.603820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1588.621341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1588.641599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1588.666451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1588.686508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1588.708819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1588.729984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1588.758422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1588.783386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1588.803979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1588.826677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1588.853418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1589.193441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.20@tcp -d 192.168.202.120@tcp -m GET -e local_timeout -r 2 [ 1589.218183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.20@tcp -d 192.168.202.120@tcp1 -m GET -e local_timeout -r 2 [ 1589.256848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.20@tcp -d 192.168.202.20@tcp -m GET -e local_timeout -r 2 [ 1589.289880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.20@tcp -d 192.168.202.20@tcp1 -m GET -e local_timeout -r 2 [ 1589.312332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.20@tcp1 -d 192.168.202.120@tcp -m GET -e local_timeout -r 2 [ 1589.337670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.20@tcp1 -d 192.168.202.120@tcp1 -m GET -e local_timeout -r 2 [ 1589.369516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.20@tcp1 -d 192.168.202.20@tcp -m GET -e local_timeout -r 2 [ 1589.390983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.20@tcp1 -d 192.168.202.20@tcp1 -m GET -e local_timeout -r 2 [ 1589.417942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.20@tcp 192.168.202.120@tcp [ 1589.433261] LNet: There was an unexpected network error while writing to 192.168.202.120: rc = -22 [ 1589.438444] LNet: Skipped 2 previous similar messages [ 1589.564600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1589.590426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1589.617496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1589.644485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1589.673307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1589.704819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1589.742352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1589.775635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1589.792509] LNet: 88973:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1589.796788] LNet: 88973:0:(api-ni.c:295:recovery_interval_set()) Skipped 1 previous similar message [ 1589.811523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1589.831946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1589.855213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1589.881304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1589.902991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1589.928724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1590.193916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.20@tcp -d 192.168.202.120@tcp -m GET -e remote_dropped -r 2 [ 1590.208465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.20@tcp -d 192.168.202.120@tcp1 -m GET -e remote_dropped -r 2 [ 1590.225621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.20@tcp -d 192.168.202.20@tcp -m GET -e remote_dropped -r 2 [ 1590.243909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.20@tcp -d 192.168.202.20@tcp1 -m GET -e remote_dropped -r 2 [ 1590.267916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.20@tcp1 -d 192.168.202.120@tcp -m GET -e remote_dropped -r 2 [ 1590.286418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.20@tcp1 -d 192.168.202.120@tcp1 -m GET -e remote_dropped -r 2 [ 1590.311378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.20@tcp1 -d 192.168.202.20@tcp -m GET -e remote_dropped -r 2 [ 1590.332130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.20@tcp1 -d 192.168.202.20@tcp1 -m GET -e remote_dropped -r 2 [ 1590.351940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.20@tcp 192.168.202.120@tcp [ 1590.361335] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.120@tcp [ 1590.509968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1590.539938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1590.559401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1590.592878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1590.623425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1590.650721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1590.667499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1590.681777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1590.697683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1590.715695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1590.733440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1590.750825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1590.766619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1590.791442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1593.809887] LNet: 89170:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1593.812896] LNetError: 89170:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1594.847148] LNet: Removed LNI 192.168.202.20@tcp [ 1594.849718] LNetError: 89170:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1594.861176] LNet: Removed LNI 192.168.202.20@tcp1 [ 1595.076606] Key type .llcrypt unregistered [ 1595.078115] Key type ._llcrypt unregistered [ 1603.792136] Key type ._llcrypt registered [ 1603.794286] Key type .llcrypt registered [ 1604.542164] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 08:23:15 (1786191795) [ 1604.758648] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1604.767637] alg: No test for adler32 (adler32-zlib) [ 1605.587305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1605.618379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1605.631515] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 1606.081629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1606.102947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.20@tcp [ 1606.117723] LNetError: 90147:0:(api-ni.c:4472:lnet_ni_set_healthv()) manually adding local NI 192.168.202.20@tcp to recovery [ 1606.201286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.202.20@tcp [ 1606.285846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1609.783495] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 08:23:20 (1786191800) [ 1609.913076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1609.927182] LNet: 90737:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1609.930823] LNetError: 90737:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1609.939842] LNet: Removed LNI 192.168.202.20@tcp [ 1609.952525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1609.989829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1610.003964] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 1610.444733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1610.461634] LNet: Added LNI 192.168.202.20@tcp1 [8/256/0/180] [ 1611.049215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.202.20@tcp [ 1611.059744] LNetError: 91003:0:(api-ni.c:4472:lnet_ni_set_healthv()) manually adding local NI 192.168.202.20@tcp to recovery [ 1611.999502] LNet: 1 local NIs in recovery (showing 1): 192.168.202.20@tcp [ 1612.001274] LNet: There was an unexpected network error while writing to 192.168.202.20: rc = -22 [ 1612.190508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.202.20@tcp [ 1617.045259] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 08:23:27 (1786191807) [ 1618.061108] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg220-server [ 1618.929628] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 08:23:29 (1786191809) [ 1619.899311] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg220-server [ 1620.784893] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 08:23:31 (1786191811) [ 1621.776839] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg220-server [ 1622.716746] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 08:23:33 (1786191813) [ 1622.855991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1622.865302] LNet: 92623:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1622.868680] LNetError: 92623:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1622.871575] LNet: Removed LNI 192.168.202.20@tcp [ 1622.878918] LNet: Removed LNI 192.168.202.20@tcp1 [ 1622.889559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1622.917926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1622.933525] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 1623.366833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1623.403799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1623.436758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1623.479597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1623.512997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1623.539913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1627.005499] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 08:23:37 (1786191817) [ 1627.195861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1627.210414] LNet: 93348:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1627.214514] LNetError: 93348:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1627.218057] LNetError: 93348:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1627.224616] LNet: Removed LNI 192.168.202.20@tcp [ 1627.233366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1627.262559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1627.278072] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 1631.172732] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 08:23:41 (1786191821) [ 1631.307904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1631.333456] LNet: 94049:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1631.343845] LNetError: 94049:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1631.356792] LNet: Removed LNI 192.168.202.20@tcp [ 1631.373549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1631.412848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1631.435487] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 1635.973420] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 08:23:46 (1786191826) [ 1636.926664] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1637.952607] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 08:23:48 (1786191828) [ 1638.221551] LNet: 95084:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1638.231753] LNetError: 95084:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1638.258657] LNet: Removed LNI 192.168.202.20@tcp [ 1638.705823] Key type .llcrypt unregistered [ 1638.707177] Key type ._llcrypt unregistered [ 1642.212557] Key type ._llcrypt registered [ 1642.214495] Key type .llcrypt registered [ 1642.293880] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1642.304843] alg: No test for adler32 (adler32-zlib) [ 1643.210854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1643.240883] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 1648.539673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.120@tcp [ 1652.001150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.202.120@tcp [ 1652.325410] LNet: 95509:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1652.331143] LNetError: 95509:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1652.343430] LNet: Removed LNI 192.168.202.20@tcp [ 1652.685812] Key type .llcrypt unregistered [ 1652.688930] Key type ._llcrypt unregistered [ 1659.948285] Key type ._llcrypt registered [ 1659.950451] Key type .llcrypt registered [ 1660.710746] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1661.706568] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1662.675178] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 08:24:13 (1786191853) [ 1663.674748] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1664.619503] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 08:24:15 (1786191855) [ 1665.730111] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg220-server [ 1666.711511] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 08:24:17 (1786191857) [ 1667.849861] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg220-server [ 1669.020482] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 08:24:19 (1786191859) [ 1669.985252] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1670.050224] alg: No test for adler32 (adler32-zlib) [ 1671.015703] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 1671.184883] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 1672.815193] Key type lgssc registered [ 1673.445449] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1686.083353] Key type lgssc unregistered [ 1686.248634] LNet: 98682:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1686.252374] LNetError: 98682:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1686.262424] LNet: Removed LNI 192.168.202.20@tcp [ 1686.725897] Key type .llcrypt unregistered [ 1686.727414] Key type ._llcrypt unregistered [ 1695.540186] Key type ._llcrypt registered [ 1695.542454] Key type .llcrypt registered [ 1696.197431] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 08:24:46 (1786191886) [ 1696.519866] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1696.527123] alg: No test for adler32 (adler32-zlib) [ 1697.371141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1697.414663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1697.444317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1697.473784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1697.494087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1701.261806] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 08:24:51 (1786191891) [ 1701.353403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1701.371514] LNet: 100089:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1701.412468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1701.492956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1701.517071] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 1702.042640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1702.061633] LNet: 100241:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1702.065655] LNetError: 100241:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1702.081827] LNet: Removed LNI 192.168.202.20@tcp [ 1702.094753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1702.126780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1702.146128] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 1702.424172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1702.448110] LNet: Removed LNI 192.168.202.20@tcp [ 1702.457585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1702.490553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1702.504220] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 1706.486443] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 08:24:57 (1786191897) [ 1706.628421] LNet: 100859:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1706.632072] LNet: 100859:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1706.638534] LNetError: 100859:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1706.652108] LNetError: 100859:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1706.679157] LNet: Removed LNI 192.168.202.20@tcp [ 1707.154802] Key type .llcrypt unregistered [ 1707.156312] Key type ._llcrypt unregistered [ 1710.987507] Key type ._llcrypt registered [ 1710.989498] Key type .llcrypt registered [ 1711.143181] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1711.153621] alg: No test for adler32 (adler32-zlib) [ 1712.064286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1712.138060] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 1712.140491] LNet: *** cfs_fail_loc=e003, val=100*** [ 1712.586701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.20@tcp [ 1712.602218] Ping reply received with mapping: [ 1712.603653] found metadata reply NI[192.168.202.20@tcp] [ 1712.605417] 14 ca a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1712.611865] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1712.613712] 5a 5a 5a 5a 5a 5a 5a 5a [ 1712.693444] LNet: 101261:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1712.697037] LNetError: 101261:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1712.712646] LNet: Removed LNI 192.168.202.20@tcp [ 1713.169499] Key type .llcrypt unregistered [ 1713.172267] Key type ._llcrypt unregistered [ 1720.448121] Key type ._llcrypt registered [ 1720.451774] Key type .llcrypt registered [ 1721.463782] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 08:25:11 (1786191911) [ 1721.953344] Key type .llcrypt unregistered [ 1721.954910] Key type ._llcrypt unregistered [ 1725.921687] Key type ._llcrypt registered [ 1725.924453] Key type .llcrypt registered [ 1726.046904] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1726.059177] alg: No test for adler32 (adler32-zlib) [ 1727.087487] LNetError: 102333:0:(api-ni.c:2849:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1727.293500] Key type .llcrypt unregistered [ 1727.294839] Key type ._llcrypt unregistered [ 1730.493493] Key type ._llcrypt registered [ 1730.496369] Key type .llcrypt registered [ 1731.293429] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 08:25:21 (1786191921) [ 1731.726587] Key type .llcrypt unregistered [ 1731.728673] Key type ._llcrypt unregistered [ 1735.756381] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1735.761180] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1738.061342] Key type ._llcrypt registered [ 1738.062874] Key type .llcrypt registered [ 1738.163512] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1738.176265] alg: No test for adler32 (adler32-zlib) [ 1739.033717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1739.074361] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1746.019861] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 08:25:36 (1786191936) [ 1746.081973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1746.098944] LNet: 104113:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1746.103396] LNetError: 104113:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1746.119915] LNet: Removed LNI 10.1.2.3@tcp [ 1746.136583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1746.603740] Key type .llcrypt unregistered [ 1746.606506] Key type ._llcrypt unregistered [ 1750.520608] Key type ._llcrypt registered [ 1750.529949] Key type .llcrypt registered [ 1750.650319] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1750.667861] alg: No test for adler32 (adler32-zlib) [ 1751.535470] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 1751.571553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.20@tcp [ 1751.648520] LNet: 104412:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1751.655685] LNetError: 104412:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1751.671199] LNet: Removed LNI 192.168.202.20@tcp [ 1752.061428] Key type .llcrypt unregistered [ 1752.067832] Key type ._llcrypt unregistered [ 1760.054891] Key type ._llcrypt registered [ 1760.057474] Key type .llcrypt registered [ 1761.047646] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 08:25:51 (1786191951) [ 1761.433593] Key type .llcrypt unregistered [ 1761.435120] Key type ._llcrypt unregistered [ 1764.719955] Key type ._llcrypt registered [ 1764.721218] Key type .llcrypt registered [ 1764.814618] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1764.826867] alg: No test for adler32 (adler32-zlib) [ 1769.068401] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 08:25:59 (1786191959) [ 1769.140209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1769.163643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1769.239128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1769.268251] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 1769.619945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1769.652779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1769.675331] LNet: Removed LNI 192.168.202.20@tcp [ 1769.838957] LNet: 106166:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1770.037456] Key type .llcrypt unregistered [ 1770.038643] Key type ._llcrypt unregistered [ 1776.994498] Key type ._llcrypt registered [ 1776.996546] Key type .llcrypt registered [ 1777.738173] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 08:26:08 (1786191968) [ 1778.057287] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1778.068792] alg: No test for adler32 (adler32-zlib) [ 1778.916153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1778.955585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1778.969397] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 1779.424647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.20@tcp [ 1779.439598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.202.20@tcp [ 1779.450035] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1779.452677] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1779.454549] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1779, 0, 0 [ 1779.457565] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1779.458772] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1779.461021] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1779.463442] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1779, 0, 0 [ 1779.466471] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1779.468286] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1779.471073] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1779.473470] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1779, 0, 1829 [ 1779.476790] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1779.478149] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1779.480103] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1779.482639] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1779, 0, 1829 [ 1779.484839] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1783.794756] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 08:26:14 (1786191974) [ 1784.223867] LNet: 107715:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1784.230191] LNetError: 107715:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1784.247292] LNet: Removed LNI 192.168.202.20@tcp [ 1784.656547] Key type .llcrypt unregistered [ 1784.658345] Key type ._llcrypt unregistered [ 1788.565579] Key type ._llcrypt registered [ 1788.566956] Key type .llcrypt registered [ 1788.652342] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1788.658080] alg: No test for adler32 (adler32-zlib) [ 1789.525150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1789.543314] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 1794.461389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.120@tcp [ 1799.751675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1799.791199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1799.819209] LNet: Added LNI 192.168.202.20@tcp1 [8/256/0/180] [ 1800.390174] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1800.488061] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1800.495733] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1800.502891] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1802.781906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1802.795635] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1808.912849] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 08:26:39 (1786191999) [ 1809.145072] LNet: 109223:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1809.149669] LNetError: 109223:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1809.156342] LNet: Removed LNI 192.168.202.20@tcp [ 1810.207142] LNet: Removed LNI 192.168.202.20@tcp1 [ 1810.208870] LNetError: 109223:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1810.217027] LNetError: 109223:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1810.229528] LNet: Removed LNI 10.1.2.3@tcp99 [ 1810.643648] Key type .llcrypt unregistered [ 1810.645025] Key type ._llcrypt unregistered [ 1814.653584] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1814.656354] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1816.999923] Key type ._llcrypt registered [ 1817.001975] Key type .llcrypt registered [ 1817.156657] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1817.174221] alg: No test for adler32 (adler32-zlib) [ 1818.047560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1818.079625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1818.094288] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 1818.625406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1818.638941] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1820.109474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 1820.135063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1820.161737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 1820.176559] LNetError: 109972:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1820.206848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1820.235764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.20@tcp [ 1820.470488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1820.513135] LNet: 109992:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1820.517376] LNetError: 109992:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1820.522247] LNet: Removed LNI 10.1.2.3@tcp [ 1821.542177] LNet: Removed LNI 192.168.202.20@tcp [ 1821.971457] Key type .llcrypt unregistered [ 1821.974196] Key type ._llcrypt unregistered [ 1828.820042] Key type ._llcrypt registered [ 1828.821672] Key type .llcrypt registered [ 1829.624586] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 08:27:00 (1786192020) [ 1829.875847] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1829.887665] alg: No test for adler32 (adler32-zlib) [ 1830.732589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1830.764549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1830.781232] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 1831.220126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.20@tcp [ 1831.258141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg220-client@tcp [ 1834.777180] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 08:27:05 (1786192025) [ 1834.880765] LNet: 111523:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1834.883611] LNetError: 111523:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1834.902240] LNet: Removed LNI 192.168.202.20@tcp [ 1835.248543] Key type .llcrypt unregistered [ 1835.250400] Key type ._llcrypt unregistered [ 1838.118444] Key type ._llcrypt registered [ 1838.120992] Key type .llcrypt registered [ 1838.224652] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1838.236655] alg: No test for adler32 (adler32-zlib) [ 1839.068961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 1839.081971] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 1839.096855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1842.116244] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 08:27:12 (1786192032) [ 1842.895359] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 1843.778444] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 08:27:14 (1786192034) [ 1844.711701] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 1845.697631] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 08:27:16 (1786192036) [ 1845.762960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1845.783920] LNet: 112943:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1845.787428] LNetError: 112943:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1845.811280] LNet: Removed LNI 192.168.202.20@tcp [ 1845.823276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1845.856762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1845.873271] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 1846.335664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1846.362464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1849.930163] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 08:27:20 (1786192040) [ 1849.997742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1850.019266] LNet: 113656:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1850.023822] LNetError: 113656:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1850.034096] LNet: Removed LNI 192.168.202.20@tcp [ 1850.045704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1850.074477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 1850.096266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1850.412293] Key type .llcrypt unregistered [ 1850.413830] Key type ._llcrypt unregistered [ 1856.161363] Key type ._llcrypt registered [ 1856.163707] Key type .llcrypt registered [ 1856.946808] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 08:27:27 (1786192047) [ 1857.252713] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1857.260984] alg: No test for adler32 (adler32-zlib) [ 1858.169848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1858.212511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1858.225408] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 1858.641825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1858.691847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.20@tcp [ 1862.819447] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 08:27:33 (1786192053) [ 1862.921787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1862.943450] LNet: 115214:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1862.946938] LNetError: 115214:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1862.961725] LNet: Removed LNI 192.168.202.20@tcp [ 1862.974569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1863.004165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 1863.018695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 1867.024474] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 08:27:37 (1786192057) [ 1867.139848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1867.159213] LNet: 115795:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1867.177380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1867.232313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1867.479915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1867.502195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1867.541607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 1867.772471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1867.791809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1867.836710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1867.856622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1867.897074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1868.072184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1868.101355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1868.134547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 1868.356638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1868.377908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1872.204795] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 08:27:42 (1786192062) [ 1872.292710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1872.306286] LNet: 116425:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1872.310562] LNet: 116425:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1872.328041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1872.366622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1872.381144] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 1872.786283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1872.798486] LNet: Added LNI 192.168.202.20@tcp2 [8/256/0/180] [ 1873.292950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1873.328410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.20@tcp [ 1873.454412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1873.472929] LNetError: 116707:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1873.476975] LNet: Removed LNI 192.168.202.20@tcp [ 1873.483635] LNet: Removed LNI 192.168.202.20@tcp2 [ 1873.495951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1873.532218] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 1873.536286] LNet: Added LNI 192.168.202.20@tcp2 [8/256/0/180] [ 1873.538426] LNet: 116713:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1873.830934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.20@tcp [ 1873.920894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1874.975404] LNet: Removed LNI 192.168.202.20@tcp [ 1874.977907] LNetError: 116751:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1874.981756] LNetError: 116751:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1874.992876] LNet: Removed LNI 192.168.202.20@tcp2 [ 1875.008615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1875.045678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1875.062622] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 1875.452820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1875.469327] LNet: Added LNI 192.168.202.20@tcp2 [8/256/0/180] [ 1875.927618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.20@tcp [ 1875.949328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1875.977076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.20@tcp [ 1876.081105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1876.092309] LNet: 117035:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1876.095527] LNet: 117035:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1877.151158] LNet: Removed LNI 192.168.202.20@tcp [ 1877.155608] LNetError: 117035:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1877.162524] LNetError: 117035:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1877.174437] LNet: Removed LNI 192.168.202.20@tcp2 [ 1877.186370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1877.218092] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 1877.222792] LNet: Added LNI 192.168.202.20@tcp2 [8/256/0/180] [ 1877.226788] LNet: 117041:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1877.478144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.20@tcp [ 1877.570551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1877.582963] LNet: Removed LNI 192.168.202.20@tcp [ 1878.629631] LNet: Removed LNI 192.168.202.20@tcp2 [ 1878.652709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1878.688377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1878.704174] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 1879.112202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1879.124730] LNet: Added LNI 192.168.202.20@tcp2 [8/256/0/180] [ 1879.531233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1879.584824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.20@tcp [ 1879.733492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1880.799378] LNet: Removed LNI 192.168.202.20@tcp [ 1880.813284] LNet: Removed LNI 192.168.202.20@tcp2 [ 1880.834086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1880.872866] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 1880.884508] LNet: Added LNI 192.168.202.20@tcp2 [8/256/0/180] [ 1880.888499] LNet: 117364:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1881.169881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.20@tcp [ 1881.309708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1881.322711] LNet: 117401:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1881.324950] LNet: 117401:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1881.328122] LNetError: 117401:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1881.331181] LNetError: 117401:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1881.334278] LNet: Removed LNI 192.168.202.20@tcp [ 1881.342302] LNet: Removed LNI 192.168.202.20@tcp2 [ 1881.366058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1881.413753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1881.427357] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 1881.885707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1881.897945] LNet: Added LNI 192.168.202.20@tcp2 [8/256/0/180] [ 1882.560486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.20@tcp [ 1882.592488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1882.646456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.20@tcp [ 1882.778118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1883.807619] LNet: Removed LNI 192.168.202.20@tcp [ 1884.844863] LNet: Removed LNI 192.168.202.20@tcp2 [ 1884.858911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1884.913894] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 1884.922303] LNet: Added LNI 192.168.202.20@tcp2 [8/256/0/180] [ 1884.928109] LNet: 117691:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1885.223757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.20@tcp [ 1885.349476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1885.363341] LNet: Removed LNI 192.168.202.20@tcp [ 1886.373705] LNet: Removed LNI 192.168.202.20@tcp2 [ 1886.402163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1886.447047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1886.466689] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 1886.900596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1886.915844] LNet: Added LNI 192.168.202.20@tcp2 [8/256/0/180] [ 1887.462546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1887.487305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1887.532555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.20@tcp [ 1887.757473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1888.799496] LNet: Removed LNI 192.168.202.20@tcp [ 1888.808923] LNet: Removed LNI 192.168.202.20@tcp2 [ 1888.819960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1888.885766] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 1888.892600] LNet: Added LNI 192.168.202.20@tcp2 [8/256/0/180] [ 1889.319485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.20@tcp [ 1889.548474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1889.567377] LNet: 118083:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1889.571884] LNet: 118083:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1889.576672] LNetError: 118083:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1889.581475] LNetError: 118083:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 1889.590868] LNet: Removed LNI 192.168.202.20@tcp [ 1889.598636] LNet: Removed LNI 192.168.202.20@tcp2 [ 1889.626846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1889.678931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1889.700064] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 1890.185359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1890.197904] LNet: Added LNI 192.168.202.20@tcp2 [8/256/0/180] [ 1890.753435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.20@tcp [ 1890.776407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1890.796676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1890.837650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.20@tcp [ 1891.086416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1892.134803] LNet: Removed LNI 192.168.202.20@tcp [ 1893.158529] LNet: Removed LNI 192.168.202.20@tcp2 [ 1893.192117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1893.255081] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 1893.259887] LNet: Added LNI 192.168.202.20@tcp2 [8/256/0/180] [ 1893.264603] LNet: 118391:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1893.269646] LNet: 118391:0:(api-ni.c:295:recovery_interval_set()) Skipped 1 previous similar message [ 1893.548504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.20@tcp [ 1898.145782] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 1899.292852] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 08:28:09 (1786192089) [ 1899.473304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1899.494543] LNet: Removed LNI 192.168.202.20@tcp [ 1900.517957] LNet: Removed LNI 192.168.202.20@tcp2 [ 1900.534715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1900.733307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1900.755684] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 1901.149624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1901.159373] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1901.729326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1901.737436] LNet: Removed LNI 10.1.2.3@tcp [ 1901.751243] LNet: Removed LNI 192.168.202.20@tcp [ 1901.763930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1901.792163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 1901.804251] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 1901.806800] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1902.104294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 1902.121892] LNet: Removed LNI 192.168.202.20@tcp [ 1902.135895] LNet: Removed LNI 10.1.2.3@tcp [ 1905.867967] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 08:28:16 (1786192096) [ 1905.977328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1905.997952] LNet: 120022:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1906.009046] LNet: 120022:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1906.031342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1906.077236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1906.089406] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 1906.592426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 1906.623602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.20@tcp [ 1906.657980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.20@tcp [ 1906.675448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.20@tcp [ 1906.695269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.20@tcp [ 1906.730555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.20@tcp [ 1906.772907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.20@tcp [ 1906.799701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.20@tcp [ 1906.823740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.20@tcp [ 1906.847661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.20@tcp [ 1906.873265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.20@tcp [ 1906.991212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 1911.137976] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 08:28:21 (1786192101) [ 1911.285801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1911.304874] LNetError: 120763:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1911.307760] LNetError: 120763:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 1911.323647] LNet: Removed LNI 192.168.202.20@tcp [ 1911.342132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1911.437742] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1911.524311] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1911.527360] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1911.530084] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1913.758552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1913.775689] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 1914.233323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1914.249175] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1915.380961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.20@tcp [ 1915.397942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 1915.436626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.202.20@tcp -m PUT -e local_timeout -r 1 [ 1915.468817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 1915.500614] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1915.502994] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1915.642867] LNetError: 120827:0:(lib-move.c:4801:lnet_parse()) 192.168.202.20@tcp, src 192.168.202.20@tcp: Dropping PUT (error -108 looking up sender) [ 1916.708125] LNet: Removed LNI 10.1.2.3@tcp [ 1917.743531] LNet: Removed LNI 192.168.202.20@tcp [ 1918.020714] Key type .llcrypt unregistered [ 1918.021753] Key type ._llcrypt unregistered [ 1924.981129] Key type ._llcrypt registered [ 1924.984699] Key type .llcrypt registered [ 1925.858435] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 08:28:36 (1786192116) [ 1926.066690] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1926.075735] alg: No test for adler32 (adler32-zlib) [ 1926.930591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1927.131755] LNet: 122142:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1927.504494] Key type .llcrypt unregistered [ 1927.505674] Key type ._llcrypt unregistered [ 1931.243325] Key type ._llcrypt registered [ 1931.245489] Key type .llcrypt registered [ 1931.423675] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1931.442364] alg: No test for adler32 (adler32-zlib) [ 1932.326745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1932.345338] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 1937.282594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.120@tcp [ 1937.407750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1937.772858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.120@tcp [ 1937.958186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1937.994036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1938.024205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1938.053527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1938.076873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1938.097656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1938.133451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1938.165575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1938.185572] LNet: 122596:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1938.200760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1938.224359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1938.243415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1938.268231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1938.292649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1938.325713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1938.778540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1938.931546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1938.960342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1938.984255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1939.004780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1939.029547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1939.055139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1939.077943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1939.108719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1939.129636] LNet: 122713:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1939.144982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1939.165207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1939.190771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1939.218667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1939.242802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1939.267932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1939.722952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.120@tcp [ 1939.891384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1939.912664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1939.941106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1939.973948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1939.995211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1940.021550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1940.043056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1940.069621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1940.098093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1940.128754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1940.152798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1940.182234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1940.208539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1940.239298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1940.711811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1940.885583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1940.906540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1940.929100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1940.955545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1940.981548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1941.009776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1941.030633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1941.048482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1941.061708] LNet: 122947:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1941.066509] LNet: 122947:0:(api-ni.c:295:recovery_interval_set()) Skipped 1 previous similar message [ 1941.078566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1941.104612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1941.139622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1941.165268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1941.193601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1941.227570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1941.496707] LNet: 122997:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1941.500844] LNetError: 122997:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1941.513894] LNet: Removed LNI 192.168.202.20@tcp [ 1941.842844] Key type .llcrypt unregistered [ 1941.845076] Key type ._llcrypt unregistered [ 1951.573140] Key type ._llcrypt registered [ 1951.582164] Key type .llcrypt registered [ 1952.479505] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 08:29:02 (1786192142) [ 1952.720974] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1952.732461] alg: No test for adler32 (adler32-zlib) [ 1953.603095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1953.846701] LNet: 123884:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1954.230433] Key type .llcrypt unregistered [ 1954.232549] Key type ._llcrypt unregistered [ 1958.340190] Key type ._llcrypt registered [ 1958.341827] Key type .llcrypt registered [ 1958.506505] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1958.518258] alg: No test for adler32 (adler32-zlib) [ 1959.388706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1959.409241] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 1964.214385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.120@tcp [ 1970.175537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1970.200415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1970.210647] LNet: Added LNI 192.168.202.20@tcp1 [8/256/0/180] [ 1970.664651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1970.929465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.120@tcp [ 1971.061647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1971.080508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1971.098929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1971.121654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1971.153084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1971.174406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1971.198205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1971.220053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1971.230662] LNet: 124538:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1971.247658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1971.267145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1971.281224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1971.302560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1971.322812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1971.344881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1971.745143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1971.922681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1971.941954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1971.963386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1971.984521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1972.009124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1972.030706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1972.047373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1972.066965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1972.086892] LNet: 124655:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1972.098419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1972.121546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1972.141678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1972.160530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1972.196613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1972.227343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1972.637384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.120@tcp [ 1972.703505] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.120@tcp [ 1972.772615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1972.792510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1972.813800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1972.829866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1972.852357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1972.884395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1972.909267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1972.938633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1972.963292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1972.988680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1973.014238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1973.030841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1973.072915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1973.089313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1973.536225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1973.709245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1973.737529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1973.760128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1973.784916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1973.812849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1973.833818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1973.855195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1973.879933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1973.896503] LNet: 124890:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1973.902364] LNet: 124890:0:(api-ni.c:295:recovery_interval_set()) Skipped 1 previous similar message [ 1973.920519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1973.942785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1973.980507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1974.011344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1974.045488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1974.066162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1977.799531] LNet: 124974:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1977.804431] LNetError: 124974:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1977.809928] LNet: Removed LNI 192.168.202.20@tcp [ 1977.819127] LNet: Removed LNI 192.168.202.20@tcp1 [ 1978.072610] Key type .llcrypt unregistered [ 1978.077787] Key type ._llcrypt unregistered [ 1987.060839] Key type ._llcrypt registered [ 1987.062644] Key type .llcrypt registered [ 1987.704928] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 08:29:38 (1786192178) [ 1987.881656] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1987.888840] alg: No test for adler32 (adler32-zlib) [ 1988.725972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1988.753549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1988.767987] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 1989.096252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1989.105949] LNet: Added LNI 192.168.202.20@tcp2 [8/256/0/180] [ 1989.488333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.20@tcp [ 1989.526419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 1990.559107] LNet: Removed LNI 192.168.202.20@tcp [ 1993.903103] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 08:29:44 (1786192184) [ 1994.783695] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 1995.724656] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 08:29:46 (1786192186) [ 1999.577507] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 08:29:50 (1786192190) [ 2003.018248] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 08:29:53 (1786192193) [ 2006.621393] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 08:29:57 (1786192197) [ 2010.145595] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 08:30:01 (1786192201) [ 2013.546444] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 08:30:04 (1786192204) [ 2016.683895] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 08:30:07 (1786192207) [ 2020.735512] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 08:30:11 (1786192211) [ 2024.365193] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 08:30:15 (1786192215) [ 2027.764923] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 08:30:18 (1786192218) [ 2031.484170] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 08:30:22 (1786192222) [ 2035.333863] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 08:30:26 (1786192226) [ 2038.873388] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 08:30:29 (1786192229) [ 2042.462411] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 08:30:33 (1786192233) [ 2046.722323] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 08:30:37 (1786192237) [ 2050.574261] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 08:30:41 (1786192241) [ 2054.799669] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 08:30:45 (1786192245) [ 2058.706837] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 08:30:49 (1786192249) [ 2058.824052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2058.833751] LNet: 136656:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2058.839407] LNetError: 136656:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2058.856663] LNet: Removed LNI 192.168.202.20@tcp2 [ 2058.870807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2058.956070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2058.973433] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 2059.522065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2059.531943] LNet: 136798:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2059.535728] LNetError: 136798:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2059.548599] LNet: Removed LNI 192.168.202.20@tcp [ 2059.560442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2059.588454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2059.603783] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 2063.348874] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 08:30:54 (1786192254) [ 2067.071321] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 08:30:57 (1786192257) [ 2070.888463] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 08:31:01 (1786192261) [ 2074.794111] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 08:31:05 (1786192265) [ 2078.993172] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 08:31:09 (1786192269) [ 2082.880241] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 08:31:13 (1786192273) [ 2086.626095] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 08:31:17 (1786192277) [ 2090.187496] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 08:31:21 (1786192281) [ 2094.007766] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 08:31:24 (1786192284) [ 2097.715583] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 08:31:28 (1786192288) [ 2101.549954] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 08:31:32 (1786192292) [ 2105.383542] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 08:31:36 (1786192296) [ 2109.094769] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 1862 sec ========== 08:31:39 (1786192299) [ 2109.251338] LNet: 144259:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2109.254632] LNetError: 144259:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2109.270101] LNet: Removed LNI 192.168.202.20@tcp [ 2109.661058] Key type .llcrypt unregistered [ 2109.663822] Key type ._llcrypt unregistered [ 2115.949571] Key type ._llcrypt registered [ 2115.951212] Key type .llcrypt registered [ 2116.262533] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2116.269924] alg: No test for adler32 (adler32-zlib) [ 2117.239626] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 2117.356845] LNet: Added LNI 192.168.202.20@tcp [8/256/0/180] [ 2118.975382] Key type lgssc registered [ 2119.506062] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2156.801038] Lustre: Mounted lustre-client - version 2.17.56_51_geba2e8d [ 2159.993750] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2163.741955] Lustre: Unmounted lustre-client [ 2175.247812] Key type lgssc unregistered [ 2175.405862] LNet: 147209:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2175.409815] LNetError: 147209:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2175.422904] LNet: Removed LNI 192.168.202.20@tcp [ 2175.829175] Key type .llcrypt unregistered [ 2175.832207] Key type ._llcrypt unregistered