[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 426307793 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895288K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003179] x2apic enabled [ 0.004007] Switched APIC routing to physical x2apic. [ 0.005015] kvm-guest: setup PV IPIs [ 0.008322] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010012] pid_max: default: 32768 minimum: 301 [ 0.011135] LSM: Security Framework initializing [ 0.012051] Yama: becoming mindful. [ 0.013033] SELinux: Initializing. [ 0.014060] *** VALIDATE selinux *** [ 0.022057] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027303] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029048] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030113] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031121] *** VALIDATE tmpfs *** [ 0.033082] *** VALIDATE proc *** [ 0.034273] *** VALIDATE cgroup *** [ 0.035010] *** VALIDATE cgroup2 *** [ 0.037128] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038167] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.042024] Spectre V2 : User space: Vulnerable [ 0.043008] Speculative Store Bypass: Vulnerable [ 0.045804] debug: unmapping init [mem 0xffffffffb1459000-0xffffffffb1460fff] [ 0.047174] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048717] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049025] ... version: 2 [ 0.050015] ... bit width: 48 [ 0.051011] ... generic registers: 4 [ 0.052013] ... value mask: 0000ffffffffffff [ 0.053015] ... max period: 00007fffffffffff [ 0.054016] ... fixed-purpose events: 3 [ 0.055013] ... event mask: 000000070000000f [ 0.056364] rcu: Hierarchical SRCU implementation. [ 0.058545] smp: Bringing up secondary CPUs ... [ 0.059607] x86: Booting SMP configuration: [ 0.060027] .... node #0, CPUs: #1 #2 #3 [ 0.063489] smp: Brought up 1 node, 4 CPUs [ 0.065015] smpboot: Max logical packages: 1 [ 0.066024] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.157000] node 0 deferred pages initialised in 89ms [ 0.160687] devtmpfs: initialized [ 0.161255] x86/mm: Memory block size: 128MB [ 0.164102] gcov: version magic: 0x41383552 [ 0.167186] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.171094] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.173427] pinctrl core: initialized pinctrl subsystem [ 0.176185] [ 0.176716] ************************************************************* [ 0.179012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.182011] ** ** [ 0.184010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.186013] ** ** [ 0.189013] ** This means that this kernel is built to expose internal ** [ 0.192014] ** IOMMU data structures, which may compromise security on ** [ 0.194010] ** your system. ** [ 0.197012] ** ** [ 0.199013] ** If you see this message and you are not debugging the ** [ 0.201010] ** kernel, report this immediately to your vendor! ** [ 0.204014] ** ** [ 0.205010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.207014] ************************************************************* [ 0.210803] NET: Registered protocol family 16 [ 0.212421] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.215065] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.218077] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.221126] cpuidle: using governor menu [ 0.222933] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.225104] PCI: Using configuration type 1 for base access [ 0.227127] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.237077] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.239044] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.243416] cryptd: max_cpu_qlen set to 1000 [ 0.246375] ACPI: Added _OSI(Module Device) [ 0.247000] ACPI: Added _OSI(Processor Device) [ 0.248017] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.250012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.255000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.262127] ACPI: Interpreter enabled [ 0.263134] ACPI: PM: (supports S0 S3 S4 S5) [ 0.265018] ACPI: Using IOAPIC for interrupt routing [ 0.267148] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.271388] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.282918] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.286036] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.288028] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.292137] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.297786] acpiphp: Slot [2] registered [ 0.299125] acpiphp: Slot [5] registered [ 0.301128] acpiphp: Slot [6] registered [ 0.302130] acpiphp: Slot [3] registered [ 0.304194] acpiphp: Slot [4] registered [ 0.305173] acpiphp: Slot [7] registered [ 0.307127] acpiphp: Slot [8] registered [ 0.309129] acpiphp: Slot [9] registered [ 0.310102] acpiphp: Slot [10] registered [ 0.311138] acpiphp: Slot [11] registered [ 0.313110] acpiphp: Slot [12] registered [ 0.315132] acpiphp: Slot [13] registered [ 0.316104] acpiphp: Slot [14] registered [ 0.318137] acpiphp: Slot [15] registered [ 0.320102] acpiphp: Slot [16] registered [ 0.321105] acpiphp: Slot [17] registered [ 0.323095] acpiphp: Slot [18] registered [ 0.324126] acpiphp: Slot [19] registered [ 0.326090] acpiphp: Slot [20] registered [ 0.327186] acpiphp: Slot [21] registered [ 0.329085] acpiphp: Slot [22] registered [ 0.331113] acpiphp: Slot [23] registered [ 0.332099] acpiphp: Slot [24] registered [ 0.334121] acpiphp: Slot [25] registered [ 0.335128] acpiphp: Slot [26] registered [ 0.337101] acpiphp: Slot [27] registered [ 0.338085] acpiphp: Slot [28] registered [ 0.340109] acpiphp: Slot [29] registered [ 0.341096] acpiphp: Slot [30] registered [ 0.342166] acpiphp: Slot [31] registered [ 0.343000] PCI host bridge to bus 0000:00 [ 0.343000] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.346027] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.349031] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.351028] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.354036] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.357030] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.358207] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.361139] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.365536] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.373016] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.378066] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.382028] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.384030] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.387023] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.389621] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.392799] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.395049] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.398938] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.403015] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.414019] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.419017] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.425627] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.431017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.436036] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.451025] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.459088] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.465016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.472015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.489017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.504332] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.506458] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.509381] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.511358] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.513168] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.518041] iommu: Default domain type: Passthrough [ 0.519396] SCSI subsystem initialized [ 0.520125] ACPI: bus type USB registered [ 0.522145] usbcore: registered new interface driver usbfs [ 0.524089] usbcore: registered new interface driver hub [ 0.526092] usbcore: registered new device driver usb [ 0.528206] pps_core: LinuxPPS API ver. 1 registered [ 0.529011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.533080] PTP clock support registered [ 0.535148] EDAC MC: Ver: 3.0.0 [ 0.537097] PCI: Using ACPI for IRQ routing [ 0.538731] NetLabel: Initializing [ 0.539011] NetLabel: domain hash size = 128 [ 0.539958] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.541082] NetLabel: unlabeled traffic allowed by default [ 0.543119] vgaarb: loaded [ 0.544243] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.546014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.553943] clocksource: Switched to clocksource kvm-clock [ 0.668342] VFS: Disk quotas dquot_6.6.0 [ 0.669764] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.671791] *** VALIDATE ramfs *** [ 0.672533] *** VALIDATE hugetlbfs *** [ 0.673627] pnp: PnP ACPI init [ 0.675277] pnp: PnP ACPI: found 6 devices [ 0.690803] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.694552] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.696574] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.698956] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.701789] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.705807] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.708857] NET: Registered protocol family 2 [ 0.710944] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.715519] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.718060] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.723332] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.726793] TCP: Hash tables configured (established 65536 bind 65536) [ 0.729731] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.732750] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.735271] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.738186] NET: Registered protocol family 1 [ 0.741089] RPC: Registered named UNIX socket transport module. [ 0.743032] RPC: Registered udp transport module. [ 0.744480] RPC: Registered tcp transport module. [ 0.746046] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.748137] NET: Registered protocol family 44 [ 0.749646] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.751525] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.753539] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.755311] PCI: CLS 0 bytes, default 64 [ 0.756809] Unpacking initramfs... [ 2.216502] debug: unmapping init [mem 0xffffa06c7cc64000-0xffffa06c7ffcffff] [ 2.220719] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.223279] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.226450] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.729622] Initialise system trusted keyrings [ 2.731396] Key type blacklist registered [ 2.733407] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.741126] zbud: loaded [ 2.744129] *** VALIDATE nfs *** [ 2.745105] *** VALIDATE nfs4 *** [ 2.746776] pstore: using deflate compression [ 2.750168] Platform Keyring initialized [ 2.862672] NET: Registered protocol family 38 [ 2.864731] Key type asymmetric registered [ 2.866373] Asymmetric key parser 'x509' registered [ 2.868305] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.871292] io scheduler mq-deadline registered [ 2.872990] io scheduler kyber registered [ 2.874434] io scheduler bfq registered [ 2.876440] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.879447] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.883304] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.886761] ACPI: Power Button [PWRF] [ 2.893233] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.900154] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.924902] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.952232] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.982165] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.987369] Non-volatile memory driver v1.3 [ 2.989196] Linux agpgart interface v0.103 [ 3.027963] virtio_blk virtio1: [vda] 146136 512-byte logical blocks (74.8 MB/71.4 MiB) [ 3.030307] vda: detected capacity change from 0 to 74821632 [ 3.050124] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.052836] vdb: detected capacity change from 0 to 1073741824 [ 3.064842] libphy: Fixed MDIO Bus: probed [ 3.071718] usbcore: registered new interface driver usbserial_generic [ 3.074573] usbserial: USB Serial support registered for generic [ 3.077339] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.081882] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.084177] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.086710] mousedev: PS/2 mouse device common for all mice [ 3.089862] rtc_cmos 00:05: RTC can wake from S4 [ 3.091938] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.095336] rtc_cmos 00:05: registered as rtc0 [ 3.099830] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.102665] intel_pstate: CPU model not supported [ 3.105762] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.106280] hid: raw HID events driver (C) Jiri Kosina [ 3.111587] usbcore: registered new interface driver usbhid [ 3.113473] usbhid: USB HID core driver [ 3.113774] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.114678] drop_monitor: Initializing network drop monitor service [ 3.119933] Initializing XFRM netlink socket [ 3.121596] NET: Registered protocol family 10 [ 3.124951] Segment Routing with IPv6 [ 3.126908] NET: Registered protocol family 17 [ 3.128799] mpls_gso: MPLS GSO support [ 3.136124] RAS: Correctable Errors collector initialized. [ 3.137853] AVX version of gcm_enc/dec engaged. [ 3.139896] AES CTR mode by8 optimization enabled [ 3.216976] sched_clock: Marking stable (3216939745, 0)->(4141148547, -924208802) [ 3.219252] registered taskstats version 1 [ 3.221561] Loading compiled-in X.509 certificates [ 3.223595] zswap: loaded using pool lzo/zbud [ 3.247934] Key type big_key registered [ 3.260304] Key type encrypted registered [ 3.261514] ima: No TPM chip found, activating TPM-bypass! [ 3.262882] ima: Allocated hash algorithm: sha1 [ 3.264074] ima: No architecture policies found [ 3.265239] evm: Initialising EVM extended attributes: [ 3.266426] evm: security.selinux [ 3.267749] evm: security.ima [ 3.269242] evm: security.capability [ 3.270598] evm: HMAC attrs: 0x1 [ 3.273060] rtc_cmos 00:05: setting system clock to 2026-08-23 04:36:38 UTC (1787459798) [ 3.279863] debug: unmapping init [mem 0xffffffffb2403000-0xffffffffb25fffff] [ 3.284048] debug: unmapping init [mem 0xffffffffb1182000-0xffffffffb1458fff] [ 3.290072] Write protecting the kernel read-only data: 28672k [ 3.292061] debug: unmapping init [mem 0xffffffffaf803000-0xffffffffaf9fffff] [ 3.293837] debug: unmapping init [mem 0xffffffffb0114000-0xffffffffb01fffff] [ 3.325848] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.332048] systemd[1]: Detected virtualization kvm. [ 3.333153] systemd[1]: Detected architecture x86-64. [ 3.334301] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.356700] systemd[1]: No hostname configured. [ 3.357726] systemd[1]: Set hostname to . [ 3.358923] random: systemd: uninitialized urandom read (16 bytes read) [ 3.360397] systemd[1]: Initializing machine ID from random generator. [ 3.426612] random: ln: uninitialized urandom read (6 bytes read) [ 3.539866] random: systemd: uninitialized urandom read (16 bytes read) [ 3.542789] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.546390] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.551405] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Control Socket. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Sockets. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... [ OK ] Reached target Slices. Starting Create Volatile Files and Directories... [ OK ] Reached target Initrd Root Device. Starting Apply Kernel Variables... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.226155] device-mapper: uevent: version 1.0.3 [ 4.229139] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.956213] virtio_net virtio0 ens2: renamed from eth0 [ 5.001794] scsi host0: ata_piix [ 5.008120] scsi host1: ata_piix [ 5.009821] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.012579] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.648394] dracut-initqueue[584]: RTNETLINK answers: File exists [ 9.775028] random: crng init done [ 9.776264] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.147063] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.344420] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.558899] SELinux: Disabled at runtime. [ 11.608668] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.612433] systemd[1]: Detected virtualization kvm. [ 11.613547] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.106411] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.109204] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.113335] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.115177] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.119226] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.125252] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.131946] systemd[1]: Mounting Huge Pages File System... Mounting Huge Pages File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Switch Root. [ OK ] Listening on Process Core Dump Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Stopped target Initrd Root File System. Mounting Kernel Debug File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-getty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypt[ 12.213583] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS ed Volumes. Starting Apply Kernel Variables... [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target rpc_pipefs.target. Mounting POSIX Message Queue File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Stopped target Initrd File Systems. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Reached target Swap. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Flush Journal to Persistent Storage... Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 12.515235] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 12.826805] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.852123] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.930301] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.953881] EDAC sbridge: Ver: 1.1.2 [ 14.047948] Key type dns_resolver registered [ 14.350386] NFS: Registering the id_resolver key type [ 14.351568] Key type id_resolver registered [ 14.352387] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Authorization Manager. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg255-client login: [ 47.056060] libcfs: loading out-of-tree module taints kernel. [ 47.527778] Key type ._llcrypt registered [ 47.537866] Key type .llcrypt registered [ 48.086064] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 48.112097] alg: No test for adler32 (adler32-zlib) [ 48.305191] hrtimer: interrupt took 6179963 ns [ 49.952937] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [ 51.179953] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 53.000737] Key type lgssc registered [ 54.920361] Lustre: Echo OBD driver; http://www.lustre.org/ [ 204.607212] Lustre: Mounted lustre-client - version 2.17.57_65_gc740cad [ 209.976309] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 227.609451] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing check_logdir /tmp/testlogs/ [ 230.374540] Lustre: lustre-OST0000-osc-ffffa06cd8dcc000: disconnect after 23s idle [ 235.071823] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing yml_node [ 239.970988] Lustre: DEBUG MARKER: Client: 2.17.57.65 [ 242.756629] Lustre: DEBUG MARKER: MDS: 2.17.57.65 [ 245.789650] Lustre: DEBUG MARKER: OSS: 2.17.57.65 [ 247.679227] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Sun Aug 23 00:40:41 EDT 2026 [ 265.267598] Lustre: Unmounted lustre-client [ 290.329349] Key type lgssc unregistered [ 290.683759] LNet: 8428:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 290.690062] LNetError: 8428:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 290.719350] LNet: Removed LNI 192.168.202.55@tcp [ 291.716648] Key type .llcrypt unregistered [ 291.718957] Key type ._llcrypt unregistered [ 321.111099] Key type ._llcrypt registered [ 321.113709] Key type .llcrypt registered [ 321.340207] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 321.352535] alg: No test for adler32 (adler32-zlib) [ 322.308083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 322.400472] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 322.429731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 327.066805] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 327.234289] LNet: 9309:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 327.242660] LNetError: 9309:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 327.296669] LNet: Removed LNI 192.168.202.55@tcp [ 327.936633] Key type .llcrypt unregistered [ 327.939959] Key type ._llcrypt unregistered [ 334.572764] Key type ._llcrypt registered [ 334.577086] Key type .llcrypt registered [ 336.157785] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 00:42:09 (1787460129) [ 336.547850] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 336.560274] alg: No test for adler32 (adler32-zlib) [ 337.547647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 337.654477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 337.691395] LNet: 9732:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 337.713804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 337.837478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 337.885716] LNet: 9739:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 345.313970] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 00:42:19 (1787460139) [ 345.619728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 345.662677] LNet: 10319:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 345.727601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 345.853350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 353.165657] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 00:42:26 (1787460146) [ 353.277825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 353.314513] LNet: 10900:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 353.351192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 353.464849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 361.061591] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 00:42:34 (1787460154) [ 361.194525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 361.242950] LNet: 11483:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 361.309534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 361.416355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 370.918216] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 00:42:44 (1787460164) [ 371.193648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 371.223057] LNet: 12074:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 371.260051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 371.421665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 372.017834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 381.269621] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 00:42:54 (1787460174) [ 381.403678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 381.464781] LNet: 12660:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 381.511371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 381.721668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 382.587672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 383.283054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 400.350646] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 00:43:13 (1787460193) [ 400.647092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 400.700180] LNet: 13266:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 400.737679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 400.871857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 410.283476] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 00:43:24 (1787460204) [ 410.400905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 410.459224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 410.572894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 410.608706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 411.052728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 411.081496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 411.437306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 411.479811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 411.961612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 411.982752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 412.387257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 412.417413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 412.989764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 413.017967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 413.468629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 413.521285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 414.184718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 414.259333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 414.725816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 414.770783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 415.185343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 415.230985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 415.712851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 415.743495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 416.194528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 416.239704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 416.681160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 416.723532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 423.000126] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 00:43:36 (1787460216) [ 423.285214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 423.328168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 423.412363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 423.457192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 429.503045] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 00:43:43 (1787460223) [ 429.692382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 429.744739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 429.807752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 429.839514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 436.137715] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 00:43:50 (1787460230) [ 436.336678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 436.374460] LNet: 15707:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 436.383927] LNet: 15707:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 436.409638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 436.477809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 436.499977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 442.541710] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 00:43:56 (1787460236) [ 442.767382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 442.819350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 442.882588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 442.911558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 449.854458] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 00:44:03 (1787460243) [ 450.045887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 450.093450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 450.222261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 450.259831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 457.393308] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 00:44:11 (1787460251) [ 457.728883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 457.775909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 457.859331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 457.910438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 464.283712] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 00:44:18 (1787460258) [ 464.627398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 464.705779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 464.823311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 465.327860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 473.030170] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 00:44:26 (1787460266) [ 473.229304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 473.310949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 473.669471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 473.992426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 481.404514] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 00:44:34 (1787460274) [ 481.633985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 481.723944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 481.936409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 482.280341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 490.321303] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 00:44:44 (1787460284) [ 490.391435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 490.440060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 491.031551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 491.386304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 497.823859] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 00:44:51 (1787460291) [ 497.951164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 498.036299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 498.139525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 498.191555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 505.077084] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 00:44:58 (1787460298) [ 505.613951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 505.638295] LNet: 21128:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 505.652347] LNet: 21128:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 505.693312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 505.795779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 505.866777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 513.261992] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 00:45:07 (1787460307) [ 513.325255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 513.381987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 513.494727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 522.355668] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 00:45:15 (1787460315) [ 522.722572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 522.813479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 522.974652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 530.548683] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 00:45:24 (1787460324) [ 530.676812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 530.750600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 530.868380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 531.332945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 537.960621] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 00:45:32 (1787460332) [ 538.142745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 538.207807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 538.338639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 538.374712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 545.474865] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 00:45:39 (1787460339) [ 545.785838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 545.850976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 545.952256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 546.013115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 552.612882] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 00:45:46 (1787460346) [ 552.716138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 552.769621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 552.870440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 552.900500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 559.147329] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 00:45:53 (1787460353) [ 559.293946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 559.355969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 559.426899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 559.458302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 566.440270] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 00:46:00 (1787460360) [ 566.701080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 566.739530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 566.801391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 566.887674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 566.910606] LNetError: 25854:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 566.949882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 566.986966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 573.494250] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 00:46:07 (1787460367) [ 573.628237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 573.687532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 573.748902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.e9qcTnvJ1x [ 573.810455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.e9qcTnvJ1x [ 573.882306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.e9qcTnvJ1x [ 580.692256] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 00:46:14 (1787460374) [ 580.781356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 580.852677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 580.936469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 580.984193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 581.031722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 587.079386] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 00:46:20 (1787460380) [ 587.410485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 587.464123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 587.593320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 587.626339] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 589.475294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.55@tcp [ 590.814970] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [ 592.863165] Key type lgssc registered [ 594.239253] Lustre: Echo OBD driver; http://www.lustre.org/ [ 598.551261] LNetError: 27615:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.202.55@tcp99 has no available nets [ 602.041918] Key type lgssc unregistered [ 602.461025] LNetError: 28727:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 602.475666] LNet: Removed LNI 192.168.202.55@tcp [ 603.079558] Key type .llcrypt unregistered [ 603.082982] Key type ._llcrypt unregistered [ 609.596411] Key type ._llcrypt registered [ 609.598782] Key type .llcrypt registered [ 611.191671] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 00:46:44 (1787460404) [ 611.830787] Key type .llcrypt unregistered [ 611.834337] Key type ._llcrypt unregistered [ 618.559157] Key type ._llcrypt registered [ 618.563041] Key type .llcrypt registered [ 618.770618] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 618.781933] alg: No test for adler32 (adler32-zlib) [ 619.765297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 619.896633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 619.935904] LNet: 29648:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 619.942598] LNet: 29648:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 619.956320] LNet: 29648:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 619.979991] LNet: Message forwarding will be enabled in 2s [ 623.071262] LNet: Message forwarding enabled [ 623.717859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 623.745260] LNet: Message forwarding will stop in 2s to 6s [ 627.175976] LNet: Message forwarding disabled [ 627.288929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 627.307955] LNet: 29738:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 627.317524] LNet: 29738:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 627.323949] LNet: 29738:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 627.337107] LNet: Message forwarding will be enabled in 2s [ 627.543493] LNet: 29751:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 628.339683] Key type .llcrypt unregistered [ 628.344625] Key type ._llcrypt unregistered [ 641.643750] Key type ._llcrypt registered [ 641.647638] Key type .llcrypt registered [ 643.045816] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 00:47:16 (1787460436) [ 643.696906] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 643.730418] alg: No test for adler32 (adler32-zlib) [ 644.654513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 644.702178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 651.346476] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 00:47:25 (1787460445) [ 651.502657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 651.548874] LNet: 31152:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 651.595256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 651.679667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 651.719304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 651.755805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 651.802950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 651.837838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 651.867722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 651.894947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 651.928439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 651.956398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 651.985912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 652.014606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 652.053163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 652.077876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 652.115236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 652.176059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 652.247738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 652.298921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 652.362832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 652.413689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 652.450642] LNetError: 31202:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 659.357250] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 00:47:32 (1787460452) [ 659.521695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 659.566242] LNet: 31768:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 659.606575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 659.710447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 666.776733] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 00:47:40 (1787460460) [ 669.276579] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg255-server [ 671.579525] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 00:47:45 (1787460465) [ 671.715681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 671.754635] LNet: 32699:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 671.782614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 671.853834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 671.885523] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 673.045257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.202.[56-60]@tcp [ 673.068903] LNet: 32838:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 673.125313] LNetError: 32708:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.57@tcp is being used as a gateway but routing feature is not turned on [ 673.148120] LNetError: 32708:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp1 through 192.168.202.57@tcp has gone from up to down [ 680.737532] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 00:47:53 (1787460473) [ 682.718562] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg255-server [ 684.515497] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 00:47:58 (1787460478) [ 684.631413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 684.693512] LNet: 33769:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 684.703070] LNetError: 33769:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 684.750783] LNet: Removed LNI 192.168.202.55@tcp [ 684.797372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 684.895214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 684.940454] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 685.699849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.202.[56-61/2]@tcp [ 685.721534] LNet: 33891:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 685.729263] LNet: 33891:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 685.752124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.202.[56-61/2]@tcp [ 685.770734] LNetError: 33778:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.56@tcp is being used as a gateway but routing feature is not turned on [ 685.779632] LNetError: 33778:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 685.801881] LNetError: 33778:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp103 through 192.168.202.56@tcp has gone from up to down [ 685.816292] LNetError: 33778:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 693.853501] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 00:48:07 (1787460487) [ 694.052677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 694.091630] LNet: 34478:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 694.103190] LNetError: 34478:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 694.136274] LNet: Removed LNI 192.168.202.55@tcp [ 694.189304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 694.373491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 694.439329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 694.490195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 694.575943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 694.677403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 694.728605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 694.794702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 694.871982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 694.915170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 695.005548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 695.156808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 695.213965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 695.311656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 695.413444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 695.474825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 695.534888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 695.631309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 695.680378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 695.766970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 695.862247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 695.945547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 696.045342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 696.137316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 696.200912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 696.289159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 696.389646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 696.428852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 696.526592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 696.674228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 696.725205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 696.788816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 696.856509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 696.886706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 696.944485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 702.634248] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 00:48:16 (1787460496) [ 704.863362] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg255-server [ 706.610516] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 00:48:20 (1787460500) [ 708.557760] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg255-server [ 710.886869] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 00:48:24 (1787460504) [ 710.993329] LNet: 35877:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 710.997809] LNet: 35877:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 711.604760] Key type .llcrypt unregistered [ 711.608886] Key type ._llcrypt unregistered [ 717.901971] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 718.219337] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 718.233160] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 718.242908] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 720.938748] Key type ._llcrypt registered [ 720.943415] Key type .llcrypt registered [ 721.157911] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 721.168875] alg: No test for adler32 (adler32-zlib) [ 722.209611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 722.274403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 722.298634] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 722.948605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 722.980974] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 725.267966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 725.309803] LNet: Removed LNI 10.1.2.3@tcp [ 725.494755] LNet: 36610:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 725.505288] LNetError: 36610:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 725.523232] LNet: Removed LNI 192.168.202.55@tcp [ 726.346518] Key type .llcrypt unregistered [ 726.356075] Key type ._llcrypt unregistered [ 738.393362] Key type ._llcrypt registered [ 738.395477] Key type .llcrypt registered [ 739.650073] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 00:48:53 (1787460533) [ 739.984590] Key type .llcrypt unregistered [ 739.986259] Key type ._llcrypt unregistered [ 745.972069] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 746.180290] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 746.198269] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 746.208784] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 748.900725] Key type ._llcrypt registered [ 748.903560] Key type .llcrypt registered [ 749.083872] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 749.101866] alg: No test for adler32 (adler32-zlib) [ 750.065300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 750.111702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 750.152847] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 750.935581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 750.966966] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 754.357652] LNet: 38106:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 754.369791] LNetError: 38106:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 754.383611] LNet: Removed LNI 10.1.2.3@tcp [ 754.396442] LNet: Removed LNI 192.168.202.55@tcp [ 755.302819] Key type .llcrypt unregistered [ 755.308253] Key type ._llcrypt unregistered [ 768.586693] Key type ._llcrypt registered [ 768.589715] Key type .llcrypt registered [ 769.831522] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 00:49:23 (1787460563) [ 770.383552] Key type .llcrypt unregistered [ 770.385958] Key type ._llcrypt unregistered [ 776.881297] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 776.995740] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 777.002584] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 777.027809] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 779.809539] Key type ._llcrypt registered [ 779.811870] Key type .llcrypt registered [ 780.005922] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 780.026701] alg: No test for adler32 (adler32-zlib) [ 781.027573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 781.065848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 781.082507] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 783.011934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 783.037991] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 784.781259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 784.796227] LNet: Removed LNI 10.1.2.3@tcp [ 784.806471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 784.841252] LNet: Removed LNI 10.1.2.31@tcp [ 784.863277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 784.883035] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 785.485896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 785.500909] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 786.188663] LNet: 40036:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 786.195863] LNetError: 40036:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 786.202599] LNet: Removed LNI 10.1.2.31@tcp [ 786.211793] LNet: Removed LNI 10.1.2.3@tcp [ 786.870774] Key type .llcrypt unregistered [ 786.875498] Key type ._llcrypt unregistered [ 799.515222] Key type ._llcrypt registered [ 799.520584] Key type .llcrypt registered [ 800.967899] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 00:49:54 (1787460594) [ 801.327535] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 801.342227] alg: No test for adler32 (adler32-zlib) [ 802.335624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 802.392492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 802.414579] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 803.157449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 803.176617] LNet: 40994:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 803.181754] LNetError: 40994:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 803.200112] LNet: Removed LNI 192.168.202.55@tcp [ 803.233126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 803.278447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.202.55@tcp [ 803.299100] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 804.003473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.202.55@tcp [ 804.032763] LNet: Removed LNI 192.168.202.55@tcp [ 804.117681] LNet: 41129:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 804.859432] Key type .llcrypt unregistered [ 804.864149] Key type ._llcrypt unregistered [ 816.746304] Key type ._llcrypt registered [ 816.751269] Key type .llcrypt registered [ 818.310252] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 00:50:11 (1787460611) [ 820.569837] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg255-server [ 822.162323] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 00:50:16 (1787460616) [ 822.813366] Key type .llcrypt unregistered [ 822.814725] Key type ._llcrypt unregistered [ 829.863755] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 829.984837] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 829.993898] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 830.006220] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 832.792546] Key type ._llcrypt registered [ 832.797082] Key type .llcrypt registered [ 832.986251] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 833.004182] alg: No test for adler32 (adler32-zlib) [ 833.951867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 834.058995] LNet: Added LNI 192.168.202.55@tcp1 [8/256/0/180] [ 834.873063] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 837.561836] LNet: 42974:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 837.570099] LNetError: 42974:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 837.578657] LNet: Removed LNI 192.168.202.55@tcp1 [ 837.593901] LNet: Removed LNI 10.1.2.3@tcp2 [ 838.392224] Key type .llcrypt unregistered [ 838.393957] Key type ._llcrypt unregistered [ 850.711754] Key type ._llcrypt registered [ 850.713435] Key type .llcrypt registered [ 851.908896] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 00:50:45 (1787460645) [ 852.286992] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 852.307570] alg: No test for adler32 (adler32-zlib) [ 853.253808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 853.334277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 853.358928] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 854.060114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 854.075643] LNet: 43948:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 854.082911] LNetError: 43948:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 854.101488] LNet: Removed LNI 192.168.202.55@tcp [ 854.119977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 854.212922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 854.242855] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 861.231655] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 00:50:55 (1787460655) [ 861.544721] LNet: 44539:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 861.560055] LNetError: 44539:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 861.579043] LNet: Removed LNI 192.168.202.55@tcp [ 862.425627] Key type .llcrypt unregistered [ 862.427463] Key type ._llcrypt unregistered [ 868.786689] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 869.053480] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 869.055850] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 869.068786] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 871.586403] Key type ._llcrypt registered [ 871.588911] Key type .llcrypt registered [ 871.800855] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 871.813890] alg: No test for adler32 (adler32-zlib) [ 872.762601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 872.835076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 872.854982] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 873.511557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 873.525951] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 875.374385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 875.391862] LNet: 45286:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 875.397799] LNetError: 45286:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 875.405610] LNet: Removed LNI 10.1.2.3@tcp [ 875.413750] LNet: Removed LNI 192.168.202.55@tcp [ 875.438468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 875.506201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 875.528687] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 875.533673] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 875.962710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 875.989739] LNet: 45313:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 876.001178] LNetError: 45313:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 876.007804] LNetError: 45313:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 876.012661] LNet: Removed LNI 10.1.2.3@tcp [ 876.028733] LNet: Removed LNI 192.168.202.55@tcp [ 876.066332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 876.156783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 876.209527] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 876.220666] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 876.633875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 876.674087] LNet: Added LNI 192.168.202.55@tcp2 [8/256/0/180] [ 877.694525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 877.712139] LNet: 45476:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 877.718927] LNetError: 45476:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 877.728266] LNetError: 45476:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 877.734168] LNet: Removed LNI 10.1.2.3@tcp [ 877.735775] LNet: Removed LNI 192.168.202.55@tcp [ 877.747258] LNet: Removed LNI 192.168.202.55@tcp2 [ 877.766785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 877.812867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 877.834520] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 877.838616] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 877.843096] LNet: Added LNI 192.168.202.55@tcp2 [8/256/0/180] [ 884.569700] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 00:51:18 (1787460678) [ 884.778535] LNet: 46073:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 884.788820] LNetError: 46073:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 884.798827] LNetError: 46073:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 884.806418] LNet: Removed LNI 10.1.2.3@tcp [ 884.808842] LNet: Removed LNI 192.168.202.55@tcp [ 884.819900] LNet: Removed LNI 192.168.202.55@tcp2 [ 885.601905] Key type .llcrypt unregistered [ 885.603755] Key type ._llcrypt unregistered [ 892.973552] Key type ._llcrypt registered [ 892.975808] Key type .llcrypt registered [ 893.340414] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 893.361488] alg: No test for adler32 (adler32-zlib) [ 894.367668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 894.428683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 894.450257] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 895.195729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 895.214498] LNet: 46472:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 895.220153] LNetError: 46472:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 895.243807] LNet: Removed LNI 192.168.202.55@tcp [ 895.276502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 895.388455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 895.418478] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 896.007696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 896.022279] LNet: 46502:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 896.030562] LNetError: 46502:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 896.042877] LNet: Removed LNI 192.168.202.55@tcp [ 896.061726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 896.119487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 896.146088] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 896.583847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 896.615928] LNet: Removed LNI 192.168.202.55@tcp [ 896.633926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 896.720711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 896.757375] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 897.403993] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 897.504380] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 897.509593] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 897.521513] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 900.123948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 900.168064] LNet: 46612:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 900.180809] LNet: 46612:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 900.185060] LNetError: 46612:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 900.194565] LNetError: 46612:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 900.227889] LNet: Removed LNI 192.168.202.55@tcp [ 900.264989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 900.380660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 900.432163] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 907.597411] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 00:51:41 (1787460701) [ 907.983444] LNet: 47211:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 908.002324] LNetError: 47211:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 908.030573] LNet: Removed LNI 192.168.202.55@tcp [ 908.778756] Key type .llcrypt unregistered [ 908.783463] Key type ._llcrypt unregistered [ 915.344368] Key type ._llcrypt registered [ 915.360573] Key type .llcrypt registered [ 915.567861] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 915.586674] alg: No test for adler32 (adler32-zlib) [ 916.520069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 916.562991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 916.581870] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 917.303706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 917.317573] LNet: 47614:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 917.323148] LNetError: 47614:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 917.336345] LNet: Removed LNI 192.168.202.55@tcp [ 917.352679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 917.416404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 917.438658] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 923.927854] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 00:51:57 (1787460717) [ 924.258947] LNet: 48207:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 924.265363] LNetError: 48207:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 924.302718] LNet: Removed LNI 192.168.202.55@tcp [ 924.916497] Key type .llcrypt unregistered [ 924.922862] Key type ._llcrypt unregistered [ 931.567878] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 931.802605] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 931.805109] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 931.813965] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 934.511912] Key type ._llcrypt registered [ 934.517909] Key type .llcrypt registered [ 934.741991] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 934.762672] alg: No test for adler32 (adler32-zlib) [ 935.727300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 935.797919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 935.819479] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 936.711386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 936.752283] LNet: 48657:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 936.765292] LNetError: 48657:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 936.779979] LNet: Removed LNI 192.168.202.55@tcp [ 936.818705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 936.888412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 936.914107] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 943.614219] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 00:52:17 (1787460737) [ 943.729189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 943.760853] LNet: 49250:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 943.773685] LNetError: 49250:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 943.797297] LNet: Removed LNI 192.168.202.55@tcp [ 943.838897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 944.046542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 950.415629] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 00:52:24 (1787460744) [ 950.737478] LNet: 49838:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 951.547523] Key type .llcrypt unregistered [ 951.549532] Key type ._llcrypt unregistered [ 959.442992] Key type ._llcrypt registered [ 959.445623] Key type .llcrypt registered [ 959.687241] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 959.717426] alg: No test for adler32 (adler32-zlib) [ 960.756228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 960.828174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 960.860420] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 962.118987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 962.167584] LNet: 50239:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 962.176065] LNetError: 50239:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 962.191527] LNet: Removed LNI 192.168.202.55@tcp [ 962.216464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 962.306978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 962.332909] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 970.064323] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 00:52:43 (1787460763) [ 970.256642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 970.281811] LNet: 50833:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 970.292710] LNetError: 50833:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 970.311543] LNet: Removed LNI 192.168.202.55@tcp [ 970.328944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 970.396809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 976.534181] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 00:52:50 (1787460770) [ 978.210993] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 979.786728] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 00:52:53 (1787460773) [ 981.295452] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 982.961848] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 00:52:57 (1787460777) [ 984.437929] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 986.037579] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 00:52:59 (1787460779) [ 987.439680] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 989.334480] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 00:53:02 (1787460782) [ 990.965301] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 993.180118] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 00:53:06 (1787460786) [ 993.289527] LNet: 52876:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 993.955237] Key type .llcrypt unregistered [ 993.958929] Key type ._llcrypt unregistered [ 1000.734337] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1000.861412] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1000.878587] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1000.886625] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1003.832942] Key type ._llcrypt registered [ 1003.835947] Key type .llcrypt registered [ 1004.106047] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1004.120845] alg: No test for adler32 (adler32-zlib) [ 1005.125253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1005.305629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1005.342455] LNet: Added LNI 192.168.202.55@tcp [8/512/0/180] [ 1005.349988] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1005.352192] LNet: 53210:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1005.479178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1005.507803] LNet: 53229:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1005.512648] LNetError: 53229:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1005.517064] LNet: Removed LNI 10.1.2.3@tcp [ 1005.518939] LNetError: 53229:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1005.529851] LNet: Removed LNI 192.168.202.55@tcp [ 1005.543299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1005.580322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1005.594200] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 1005.632556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1005.643568] LNet: 53253:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1005.646824] LNetError: 53253:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1005.661656] LNet: Removed LNI 192.168.202.55@tcp [ 1005.682251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1005.750548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1005.764912] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 1005.767509] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1005.768724] LNet: 53261:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1005.875520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1005.915719] LNet: 53279:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1005.920464] LNetError: 53279:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1005.928307] LNet: Removed LNI 10.1.2.3@tcp [ 1005.930330] LNetError: 53279:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1005.960317] LNet: Removed LNI 192.168.202.55@tcp [ 1005.992391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1006.053605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1006.079133] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 1006.082443] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1006.084593] LNet: 53287:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1006.372130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1006.413561] LNet: 53313:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1006.425581] LNetError: 53313:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1006.441763] LNet: Removed LNI 10.1.2.3@tcp [ 1006.444550] LNetError: 53313:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1006.464686] LNet: Removed LNI 192.168.202.55@tcp [ 1006.533540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1006.589655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1006.610491] LNet: Added LNI 192.168.202.55@tcp [8/256/0/182] [ 1006.614590] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1006.678064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1006.692162] LNet: 53341:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1006.699840] LNetError: 53341:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1006.706197] LNet: Removed LNI 10.1.2.3@tcp [ 1006.710475] LNetError: 53341:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1006.724482] LNet: Removed LNI 192.168.202.55@tcp [ 1006.746586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1006.799877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1006.815878] LNet: Added LNI 192.168.202.55@tcp [8/256/0/182] [ 1006.818641] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1006.821411] LNet: 53349:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1006.864460] LNet: Removed LNI 10.1.2.3@tcp [ 1006.872596] LNet: Removed LNI 192.168.202.55@tcp [ 1007.309559] Key type .llcrypt unregistered [ 1007.311539] Key type ._llcrypt unregistered [ 1019.677458] Key type ._llcrypt registered [ 1019.683701] Key type .llcrypt registered [ 1021.552854] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 00:53:34 (1787460814) [ 1023.167938] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 1025.402386] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 00:53:39 (1787460819) [ 1025.890672] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1025.906153] alg: No test for adler32 (adler32-zlib) [ 1026.848861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1026.903339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1026.931884] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 1028.082153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.202.55 [ 1028.100158] LNet: 54625:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1028.106166] LNet: Removed LNI 192.168.202.55@tcp [ 1034.485675] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 00:53:48 (1787460828) [ 1034.593661] LNet: 55188:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1034.606584] LNetError: 55188:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1034.642592] LNet: Removed LNI 192.168.202.55@tcp [ 1035.454628] Key type .llcrypt unregistered [ 1035.458684] Key type ._llcrypt unregistered [ 1043.113804] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1043.305770] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1045.963243] Key type ._llcrypt registered [ 1045.966153] Key type .llcrypt registered [ 1046.166260] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1046.183567] alg: No test for adler32 (adler32-zlib) [ 1047.689816] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1055.229624] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 00:54:08 (1787460848) [ 1055.298904] LNet: 56103:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1055.304403] LNetError: 56103:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1055.315840] LNet: Removed LNI 10.1.2.3@tcp [ 1056.171429] Key type .llcrypt unregistered [ 1056.173993] Key type ._llcrypt unregistered [ 1062.930409] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1063.148861] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1063.158687] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1063.165626] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1066.045444] Key type ._llcrypt registered [ 1066.047174] Key type .llcrypt registered [ 1066.167408] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1066.184543] alg: No test for adler32 (adler32-zlib) [ 1067.162341] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1073.185748] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 00:54:27 (1787460867) [ 1074.860486] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1076.765067] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 00:54:30 (1787460870) [ 1076.895883] LNet: 57291:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1076.911171] LNetError: 57291:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1076.952199] LNet: Removed LNI 10.1.2.3@tcp [ 1077.884662] Key type .llcrypt unregistered [ 1077.887283] Key type ._llcrypt unregistered [ 1084.725043] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1084.940565] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1084.943168] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1084.945955] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1087.613398] Key type ._llcrypt registered [ 1087.615237] Key type .llcrypt registered [ 1087.799166] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1087.826629] alg: No test for adler32 (adler32-zlib) [ 1088.846249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1088.926461] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1099.742948] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 00:54:53 (1787460893) [ 1100.383595] LNet: 58535:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1100.391906] LNetError: 58535:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1100.436353] LNet: Removed LNI 10.1.2.3@tcp [ 1101.187169] Key type .llcrypt unregistered [ 1101.196333] Key type ._llcrypt unregistered [ 1107.709744] Key type ._llcrypt registered [ 1107.713660] Key type .llcrypt registered [ 1107.888755] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1107.910959] alg: No test for adler32 (adler32-zlib) [ 1108.832257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1108.895228] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 1115.844562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.155@tcp [ 1116.399479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.155@tcp -m GET -e local_interrupt -r 1 [ 1116.429139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.55@tcp -m GET -e local_interrupt -r 1 [ 1116.455558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.155@tcp [ 1116.469131] LNet: There was an unexpected network error while writing to 192.168.202.155: rc = -22 [ 1116.665229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1116.689606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1116.715633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1116.745525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1116.769697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1116.795553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1116.820227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1116.851906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1116.866238] LNet: 58977:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1116.884466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1116.912430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1116.953312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1116.989849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1117.035117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1117.068716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1117.087360] LNet: 1 local NIs in recovery (showing 1): 192.168.202.55@tcp [ 1117.089809] LNet: There was an unexpected network error while writing to 192.168.202.55: rc = -22 [ 1117.103436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1117.581515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp -d 192.168.202.155@tcp -m GET -e local_dropped -r 1 [ 1117.612850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp -d 192.168.202.55@tcp -m GET -e local_dropped -r 1 [ 1117.645321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.155@tcp [ 1117.846470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1117.870986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1117.902414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1117.934329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1117.960900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1117.993895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1118.035433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1118.066450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1118.081977] LNet: 59086:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1118.104665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1118.127691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1118.162640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1118.210373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1118.252431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1118.321968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1118.367854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1118.942842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.155@tcp -m GET -e local_aborted -r 1 [ 1118.985184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.55@tcp -m GET -e local_aborted -r 1 [ 1119.025869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.155@tcp [ 1119.043796] LNet: There was an unexpected network error while writing to 192.168.202.155: rc = -22 [ 1119.048278] LNet: Skipped 1 previous similar message [ 1119.324428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1119.374444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1119.420855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1119.467390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1119.501576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1119.539730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1119.570810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1119.591637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1119.606737] LNet: 59196:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1119.620167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1119.649877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1119.679390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1119.708148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1119.739200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1119.768896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1119.798426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1120.275699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp -d 192.168.202.155@tcp -m GET -e local_no_route -r 1 [ 1120.307556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.55@tcp -m GET -e local_no_route -r 1 [ 1120.340519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.155@tcp [ 1120.619024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1120.641492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1120.667751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1120.690656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1120.712784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1120.741395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1120.771128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1120.795358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1120.825819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1120.852248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1120.879748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1120.900265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1120.927661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1120.956546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1120.992407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1121.540800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.155@tcp -m GET -e local_timeout -r 1 [ 1121.628611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.55@tcp -m GET -e local_timeout -r 1 [ 1121.677634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.155@tcp [ 1121.707801] LNet: There was an unexpected network error while writing to 192.168.202.155: rc = -22 [ 1121.712301] LNet: Skipped 2 previous similar messages [ 1122.033677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1122.076966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1122.144958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1122.172450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1122.216558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1122.267972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1122.321895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1122.362662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1122.377699] LNet: 59414:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1122.388834] LNet: 59414:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1122.418957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1122.487598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1122.527819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1122.564739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1122.612815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1122.651551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1122.704239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1123.362495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp -d 192.168.202.155@tcp -m GET -e local_error -r 1 [ 1123.395643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.55@tcp -m GET -e local_error -r 1 [ 1123.426626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.155@tcp [ 1123.679781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1123.712238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1123.738957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1123.806681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1123.845267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1123.870793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1123.890665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1123.910247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1123.935599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1123.963659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1123.999891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1124.036415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1124.072187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1124.110743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1124.145584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1124.470823] LNet: 59568:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1124.477361] LNetError: 59568:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1124.507878] LNet: Removed LNI 192.168.202.55@tcp [ 1125.020837] Key type .llcrypt unregistered [ 1125.024201] Key type ._llcrypt unregistered [ 1142.502158] Key type ._llcrypt registered [ 1142.506189] Key type .llcrypt registered [ 1143.936175] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 00:55:37 (1787460937) [ 1144.951703] Key type .llcrypt unregistered [ 1144.955403] Key type ._llcrypt unregistered [ 1152.979825] Key type ._llcrypt registered [ 1152.984312] Key type .llcrypt registered [ 1153.261977] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1153.300577] alg: No test for adler32 (adler32-zlib) [ 1154.231504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1154.254236] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 1163.652607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.155@tcp [ 1175.673926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1175.724694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1175.771552] LNet: Added LNI 192.168.202.55@tcp1 [8/256/0/180] [ 1177.289741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp -d 192.168.202.155@tcp -m GET -e local_interrupt -r 1 [ 1177.325030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.155@tcp1 -m GET -e local_interrupt -r 1 [ 1177.364793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp -d 192.168.202.55@tcp -m GET -e local_interrupt -r 1 [ 1177.403811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.55@tcp1 -m GET -e local_interrupt -r 1 [ 1177.434984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp1 -d 192.168.202.155@tcp -m GET -e local_interrupt -r 1 [ 1177.477471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp1 -d 192.168.202.155@tcp1 -m GET -e local_interrupt -r 1 [ 1177.518833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp1 -d 192.168.202.55@tcp -m GET -e local_interrupt -r 1 [ 1177.563749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp1 -d 192.168.202.55@tcp1 -m GET -e local_interrupt -r 1 [ 1177.641309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.155@tcp [ 1177.661189] LNet: There was an unexpected network error while writing to 192.168.202.155: rc = -22 [ 1177.666870] LNet: 1 local NIs in recovery (showing 1): 192.168.202.55@tcp [ 1177.870778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1177.902963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1177.936721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1177.978407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1178.009287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1178.051676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1178.094633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1178.134768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1178.176462] LNet: 61041:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1178.242948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1178.308448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1178.335721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1178.367771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1178.408305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1178.465645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1178.507934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1179.254256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp -d 192.168.202.155@tcp -m GET -e local_dropped -r 1 [ 1179.289245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.155@tcp1 -m GET -e local_dropped -r 1 [ 1179.335279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.55@tcp -m GET -e local_dropped -r 1 [ 1179.402833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.55@tcp1 -m GET -e local_dropped -r 1 [ 1179.464780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp1 -d 192.168.202.155@tcp -m GET -e local_dropped -r 1 [ 1179.515072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp1 -d 192.168.202.155@tcp1 -m GET -e local_dropped -r 1 [ 1179.567226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp1 -d 192.168.202.55@tcp -m GET -e local_dropped -r 1 [ 1179.615876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp1 -d 192.168.202.55@tcp1 -m GET -e local_dropped -r 1 [ 1179.663744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.155@tcp [ 1179.703494] LNet: There was an unexpected network error while writing to 192.168.202.155: rc = -22 [ 1179.712026] LNet: Skipped 2 previous similar messages [ 1180.041318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1180.090920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1180.163412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1180.225867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1180.289906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1180.317501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1180.358650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1180.394132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1180.436114] LNet: 61163:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1180.451669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1180.491724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1180.530885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1180.595169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1180.650927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1180.689644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1180.729976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1181.386476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.155@tcp -m GET -e local_aborted -r 1 [ 1181.431956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.155@tcp1 -m GET -e local_aborted -r 1 [ 1181.464648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp -d 192.168.202.55@tcp -m GET -e local_aborted -r 1 [ 1181.530677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.55@tcp1 -m GET -e local_aborted -r 1 [ 1181.583424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp1 -d 192.168.202.155@tcp -m GET -e local_aborted -r 1 [ 1181.624090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp1 -d 192.168.202.155@tcp1 -m GET -e local_aborted -r 1 [ 1181.659798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp1 -d 192.168.202.55@tcp -m GET -e local_aborted -r 1 [ 1181.706322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp1 -d 192.168.202.55@tcp1 -m GET -e local_aborted -r 1 [ 1181.743784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.155@tcp [ 1181.768912] LNet: There was an unexpected network error while writing to 192.168.202.155: rc = -22 [ 1181.782530] LNet: Skipped 2 previous similar messages [ 1182.052696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1182.098967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1182.161776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1182.215940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1182.250967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1182.279549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1182.304266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1182.340777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1182.357308] LNet: 61284:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1182.381298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1182.421247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1182.450330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1182.496701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1182.556078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1182.619316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1182.681356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1183.265881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.155@tcp -m GET -e local_no_route -r 1 [ 1183.303739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.155@tcp1 -m GET -e local_no_route -r 1 [ 1183.339546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.55@tcp -m GET -e local_no_route -r 1 [ 1183.381506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp -d 192.168.202.55@tcp1 -m GET -e local_no_route -r 1 [ 1183.411887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp1 -d 192.168.202.155@tcp -m GET -e local_no_route -r 1 [ 1183.483833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp1 -d 192.168.202.155@tcp1 -m GET -e local_no_route -r 1 [ 1183.542497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp1 -d 192.168.202.55@tcp -m GET -e local_no_route -r 1 [ 1183.581951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp1 -d 192.168.202.55@tcp1 -m GET -e local_no_route -r 1 [ 1183.616721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.155@tcp [ 1183.947885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1183.996721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1184.028681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1184.092757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1184.160344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1184.222918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1184.266499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1184.286092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1184.314164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1184.336533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1184.364883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1184.416692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1184.458161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1184.495705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1184.535753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1185.357784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.155@tcp -m GET -e local_timeout -r 1 [ 1185.406721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp -d 192.168.202.155@tcp1 -m GET -e local_timeout -r 1 [ 1185.460487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.55@tcp -m GET -e local_timeout -r 1 [ 1185.548121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp -d 192.168.202.55@tcp1 -m GET -e local_timeout -r 1 [ 1185.642659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp1 -d 192.168.202.155@tcp -m GET -e local_timeout -r 1 [ 1185.725973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp1 -d 192.168.202.155@tcp1 -m GET -e local_timeout -r 1 [ 1185.763781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp1 -d 192.168.202.55@tcp -m GET -e local_timeout -r 1 [ 1185.814280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp1 -d 192.168.202.55@tcp1 -m GET -e local_timeout -r 1 [ 1185.857689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.155@tcp [ 1185.878071] LNet: There was an unexpected network error while writing to 192.168.202.155: rc = -22 [ 1185.889387] LNet: Skipped 5 previous similar messages [ 1186.226710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1186.292182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1186.347617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1186.409630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1186.479972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1186.561157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1186.640214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1186.741660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1186.779988] LNet: 61527:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1186.803534] LNet: 61527:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1186.874566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1186.939885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1187.058556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1187.155552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1187.238727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1187.316981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1187.364455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1188.189161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp -d 192.168.202.155@tcp -m GET -e local_error -r 1 [ 1188.258786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp -d 192.168.202.155@tcp1 -m GET -e local_error -r 1 [ 1188.321918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.55@tcp -m GET -e local_error -r 1 [ 1188.393098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp -d 192.168.202.55@tcp1 -m GET -e local_error -r 1 [ 1188.466192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp1 -d 192.168.202.155@tcp -m GET -e local_error -r 1 [ 1188.515245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp1 -d 192.168.202.155@tcp1 -m GET -e local_error -r 1 [ 1188.572059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp1 -d 192.168.202.55@tcp -m GET -e local_error -r 1 [ 1188.660848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp1 -d 192.168.202.55@tcp1 -m GET -e local_error -r 1 [ 1188.708636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.155@tcp [ 1188.998885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1189.027701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1189.063659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1189.094950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1189.124255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1189.166551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1189.204846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1189.235802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1189.267542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1189.319823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1189.358494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1189.402259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1189.444692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1189.477518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1189.520334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1195.282847] LNet: 61728:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1195.290224] LNetError: 61728:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1196.319157] LNet: Removed LNI 192.168.202.55@tcp [ 1196.323598] LNetError: 61728:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1196.343589] LNet: Removed LNI 192.168.202.55@tcp1 [ 1196.850745] Key type .llcrypt unregistered [ 1196.853760] Key type ._llcrypt unregistered [ 1214.131863] Key type ._llcrypt registered [ 1214.135530] Key type .llcrypt registered [ 1215.490932] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 00:56:49 (1787461009) [ 1216.598251] Key type .llcrypt unregistered [ 1216.605348] Key type ._llcrypt unregistered [ 1224.221829] Key type ._llcrypt registered [ 1224.223332] Key type .llcrypt registered [ 1224.390031] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1224.418879] alg: No test for adler32 (adler32-zlib) [ 1225.331920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1225.357885] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 1234.919989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.155@tcp [ 1235.955354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.155@tcp -m GET -e remote_dropped -r 1 [ 1236.009410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp -d 192.168.202.55@tcp -m GET -e remote_dropped -r 1 [ 1236.050629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.155@tcp [ 1236.066498] LNet: There was an unexpected network error while writing to 192.168.202.155: rc = -22 [ 1236.272980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1236.311080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1236.344269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1236.365917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1236.397792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1236.433773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1236.458688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1236.507845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1236.541398] LNet: 62982:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1236.596408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1236.653335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1236.705728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1236.770245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1236.821052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1236.859798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1236.883920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1237.435390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp -d 192.168.202.155@tcp -m GET -e remote_error -r 1 [ 1237.503909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.55@tcp -m GET -e remote_error -r 1 [ 1237.539803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.155@tcp [ 1237.556889] LNet: There was an unexpected network error while writing to 192.168.202.155: rc = -22 [ 1237.760393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1237.791710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1237.813930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1237.836944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1237.858425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1237.884573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1237.919683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1237.952566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1237.967623] LNet: 63091:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1237.982181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1238.006874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1238.034381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1238.059511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1238.083962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1238.107745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1238.133711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1238.743613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp -d 192.168.202.155@tcp -m GET -e remote_timeout -r 1 [ 1238.796660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.55@tcp -m GET -e remote_timeout -r 1 [ 1238.832249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.155@tcp [ 1238.866592] LNet: There was an unexpected network error while writing to 192.168.202.155: rc = -22 [ 1239.322391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1239.401262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1239.463142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1239.526243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1239.592483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1239.657789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1239.735552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1239.762498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1239.772842] LNet: 63200:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1239.782361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1239.802694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1239.834115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1239.865263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1239.900210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1239.921170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1239.939741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1240.446551] LNet: 63245:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1240.456413] LNetError: 63245:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1240.475500] LNet: Removed LNI 192.168.202.55@tcp [ 1241.124479] Key type .llcrypt unregistered [ 1241.136189] Key type ._llcrypt unregistered [ 1258.587876] Key type ._llcrypt registered [ 1258.590037] Key type .llcrypt registered [ 1259.702732] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 00:57:33 (1787461053) [ 1260.862430] Key type .llcrypt unregistered [ 1260.865167] Key type ._llcrypt unregistered [ 1267.397621] Key type ._llcrypt registered [ 1267.407523] Key type .llcrypt registered [ 1267.698932] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1267.734248] alg: No test for adler32 (adler32-zlib) [ 1268.799630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1268.864688] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 1277.272410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.155@tcp [ 1286.668149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1286.753159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1286.794607] LNet: Added LNI 192.168.202.55@tcp1 [8/256/0/180] [ 1288.320315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.155@tcp -m GET -e remote_dropped -r 1 [ 1288.351051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp -d 192.168.202.155@tcp1 -m GET -e remote_dropped -r 1 [ 1288.384735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp -d 192.168.202.55@tcp -m GET -e remote_dropped -r 1 [ 1288.422573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.55@tcp1 -m GET -e remote_dropped -r 1 [ 1288.455675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp1 -d 192.168.202.155@tcp -m GET -e remote_dropped -r 1 [ 1288.487817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp1 -d 192.168.202.155@tcp1 -m GET -e remote_dropped -r 1 [ 1288.519361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp1 -d 192.168.202.55@tcp -m GET -e remote_dropped -r 1 [ 1288.556849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp1 -d 192.168.202.55@tcp1 -m GET -e remote_dropped -r 1 [ 1288.583330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.155@tcp [ 1288.601318] LNet: There was an unexpected network error while writing to 192.168.202.155: rc = -22 [ 1288.607721] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.155@tcp [ 1288.859955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1288.900972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1288.929388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1288.968781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1289.000190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1289.043623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1289.072620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1289.121974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1289.150976] LNet: 64715:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1289.169920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1289.208248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1289.238981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1289.279088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1289.324261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1289.370210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1289.416156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1289.452311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1290.142281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp -d 192.168.202.155@tcp -m GET -e remote_error -r 1 [ 1290.220439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.155@tcp1 -m GET -e remote_error -r 1 [ 1290.255988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.55@tcp -m GET -e remote_error -r 1 [ 1290.284948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp -d 192.168.202.55@tcp1 -m GET -e remote_error -r 1 [ 1290.316181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp1 -d 192.168.202.155@tcp -m GET -e remote_error -r 1 [ 1290.347411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp1 -d 192.168.202.155@tcp1 -m GET -e remote_error -r 1 [ 1290.381248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp1 -d 192.168.202.55@tcp -m GET -e remote_error -r 1 [ 1290.409819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp1 -d 192.168.202.55@tcp1 -m GET -e remote_error -r 1 [ 1290.439631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.155@tcp [ 1290.458177] LNet: There was an unexpected network error while writing to 192.168.202.155: rc = -22 [ 1290.466873] LNet: Skipped 2 previous similar messages [ 1290.657512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1290.678766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1290.709873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1290.738599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1290.772457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1290.818456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1290.848476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1290.886130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1290.906455] LNet: 64839:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1290.941047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1290.994428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1291.028279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1291.076552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1291.105167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1291.158761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1291.200506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1291.240594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1291.784696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.155@tcp -m GET -e remote_timeout -r 1 [ 1291.804248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.155@tcp1 -m GET -e remote_timeout -r 1 [ 1291.832987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.55@tcp -m GET -e remote_timeout -r 1 [ 1291.882551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp -d 192.168.202.55@tcp1 -m GET -e remote_timeout -r 1 [ 1291.953988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp1 -d 192.168.202.155@tcp -m GET -e remote_timeout -r 1 [ 1291.997513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp1 -d 192.168.202.155@tcp1 -m GET -e remote_timeout -r 1 [ 1292.032930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp1 -d 192.168.202.55@tcp -m GET -e remote_timeout -r 1 [ 1292.063587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp1 -d 192.168.202.55@tcp1 -m GET -e remote_timeout -r 1 [ 1292.111073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.155@tcp [ 1292.131891] LNet: There was an unexpected network error while writing to 192.168.202.155: rc = -22 [ 1292.344784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1292.369850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1292.399258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1292.427522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1292.462523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1292.498474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1292.546561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1292.582596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1292.600476] LNet: 64962:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1292.617728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1292.641830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1292.671902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1292.702740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1292.739701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1292.774644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1292.802950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1292.837518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1298.354267] LNet: 65044:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1298.362814] LNetError: 65044:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1298.376113] LNet: Removed LNI 192.168.202.55@tcp [ 1298.390636] LNet: Removed LNI 192.168.202.55@tcp1 [ 1298.743830] Key type .llcrypt unregistered [ 1298.747673] Key type ._llcrypt unregistered [ 1316.323340] Key type ._llcrypt registered [ 1316.325518] Key type .llcrypt registered [ 1317.454125] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 00:58:31 (1787461111) [ 1318.929784] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1321.247598] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 00:58:34 (1787461114) [ 1322.653634] Key type .llcrypt unregistered [ 1322.655875] Key type ._llcrypt unregistered [ 1330.144415] Key type ._llcrypt registered [ 1330.146942] Key type .llcrypt registered [ 1330.354671] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1330.366586] alg: No test for adler32 (adler32-zlib) [ 1331.344595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1331.391375] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 1339.791796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.155@tcp [ 1340.633991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp -d 192.168.202.155@tcp -m GET -e network_timeout -r 1 [ 1340.670551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp -d 192.168.202.55@tcp -m GET -e network_timeout -r 1 [ 1340.717817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.155@tcp [ 1340.769737] LNet: There was an unexpected network error while writing to 192.168.202.155: rc = -22 [ 1341.083556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1341.127757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1341.155114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1341.181266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1341.222787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1341.263280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1341.296767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1341.320642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1341.345868] LNet: 66593:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1341.393749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1341.456354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1341.490825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1341.527734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1341.611408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1341.655834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1341.663274] LNet: 1 local NIs in recovery (showing 1): 192.168.202.55@tcp [ 1341.665271] LNet: There was an unexpected network error while writing to 192.168.202.55: rc = -22 [ 1341.995172] LNet: 66636:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1342.001904] LNetError: 66636:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1343.079964] LNet: Removed LNI 192.168.202.55@tcp [ 1343.689337] Key type .llcrypt unregistered [ 1343.693039] Key type ._llcrypt unregistered [ 1365.398071] Key type ._llcrypt registered [ 1365.407272] Key type .llcrypt registered [ 1365.794426] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1365.829640] alg: No test for adler32 (adler32-zlib) [ 1366.795331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1366.850679] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 1374.320686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.155@tcp [ 1383.388288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1383.463211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1383.492471] LNet: Added LNI 192.168.202.55@tcp1 [8/256/0/180] [ 1384.706937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp -d 192.168.202.155@tcp -m GET -e network_timeout -r 1 [ 1384.740639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp -d 192.168.202.155@tcp1 -m GET -e network_timeout -r 1 [ 1384.774802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp -d 192.168.202.55@tcp -m GET -e network_timeout -r 1 [ 1384.810903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.55@tcp1 -m GET -e network_timeout -r 1 [ 1384.863912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp1 -d 192.168.202.155@tcp -m GET -e network_timeout -r 1 [ 1384.896630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp1 -d 192.168.202.155@tcp1 -m GET -e network_timeout -r 1 [ 1384.929820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp1 -d 192.168.202.55@tcp -m GET -e network_timeout -r 1 [ 1384.968300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp1 -d 192.168.202.55@tcp1 -m GET -e network_timeout -r 1 [ 1385.000923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.155@tcp [ 1385.025792] LNet: There was an unexpected network error while writing to 192.168.202.155: rc = -22 [ 1385.248753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1385.274612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1385.303219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1385.313145] LNet: 1 local NIs in recovery (showing 1): 192.168.202.55@tcp [ 1385.317369] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.155@tcp [ 1385.337803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1385.366840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1385.396803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1385.427369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1385.455703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1385.475300] LNet: 67520:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1385.491835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1385.514278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1385.539909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1385.564559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1385.592593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1385.618464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1390.351156] LNet: There was an unexpected network error while writing to 192.168.202.155: rc = -22 [ 1390.755641] LNet: 67598:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1390.769059] LNetError: 67598:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1390.772908] LNet: Removed LNI 192.168.202.55@tcp [ 1390.797627] LNet: Removed LNI 192.168.202.55@tcp1 [ 1391.250826] Key type .llcrypt unregistered [ 1391.254863] Key type ._llcrypt unregistered [ 1408.371315] Key type ._llcrypt registered [ 1408.372717] Key type .llcrypt registered [ 1409.411652] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 01:00:03 (1787461203) [ 1409.837667] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1409.850608] alg: No test for adler32 (adler32-zlib) [ 1410.714574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1410.753481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1410.772947] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 1411.594485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1411.628694] LNet: Added LNI 192.168.202.55@tcp1 [8/256/0/180] [ 1412.330534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.55@tcp [ 1412.381972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1412.422684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1412.447699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1412.498934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1412.554971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1412.593416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.55@tcp [ 1412.619759] LNetError: 68698:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.202.55@tcp to recovery [ 1412.769620] LNet: 1 local NIs in recovery (showing 1): 192.168.202.55@tcp [ 1412.770356] LNet: There was an unexpected network error while writing to 192.168.202.55: rc = -22 [ 1414.815873] LNet: There was an unexpected network error while writing to 192.168.202.55: rc = -22 [ 1415.161388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1418.913056] LNet: There was an unexpected network error while writing to 192.168.202.55: rc = -22 [ 1419.911509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1419.939458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1419.992783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1420.012435] LNet: 68805:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1420.020714] LNetError: 68805:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1420.029464] LNet: Removed LNI 192.168.202.55@tcp [ 1420.048581] LNet: Removed LNI 192.168.202.55@tcp1 [ 1420.067768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1420.166898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1420.198828] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 1420.889425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1420.913423] LNet: Added LNI 192.168.202.55@tcp1 [8/256/0/180] [ 1421.901619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.55@tcp [ 1421.931488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1421.987777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1422.019675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1422.052292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1422.083613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1422.117439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1422.142979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.55@tcp [ 1422.158834] LNetError: 69077:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.202.55@tcp to recovery [ 1422.181923] LNet: There was an unexpected network error while writing to 192.168.202.55: rc = -22 [ 1424.555653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1428.320924] LNet: There was an unexpected network error while writing to 192.168.202.55: rc = -22 [ 1428.324443] LNet: Skipped 1 previous similar message [ 1432.734302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1432.808171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1432.887713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1440.513108] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 01:00:34 (1787461234) [ 1440.721381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1440.732251] LNet: 69784:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1440.736489] LNetError: 69784:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1440.739573] LNetError: 69784:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1440.744829] LNet: Removed LNI 192.168.202.55@tcp [ 1440.776219] LNet: Removed LNI 192.168.202.55@tcp1 [ 1440.822907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1440.857112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1440.884276] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 1441.619865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1441.639729] LNet: Added LNI 192.168.202.55@tcp1 [8/256/0/180] [ 1442.469252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.55@tcp [ 1442.538923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1442.564931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1442.592234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1442.627858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1442.667373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1442.696457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.55@tcp --health 0 [ 1442.870172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1442.914296] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.55@tcp [ 1443.944218] LNet: There was an unexpected network error while writing to 192.168.202.55: rc = -22 [ 1443.968665] LNet: Skipped 1 previous similar message [ 1457.804613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1458.110534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1458.153981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1458.202146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.55@tcp --health 500 [ 1458.474504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1465.654309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1465.759617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1465.809376] LNet: 70268:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1465.817698] LNetError: 70268:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1465.830976] LNetError: 70268:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1465.847705] LNet: Removed LNI 192.168.202.55@tcp [ 1465.859559] LNet: Removed LNI 192.168.202.55@tcp1 [ 1465.955247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1466.079951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1466.115881] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 1467.657566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1467.698394] LNet: Added LNI 192.168.202.55@tcp1 [8/256/0/180] [ 1469.210951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.55@tcp [ 1469.291447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1469.433116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1469.480328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1469.551433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1469.592860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1469.616986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1469.640817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.55@tcp --health 0 [ 1470.111442] LNet: There was an unexpected network error while writing to 192.168.202.55: rc = -22 [ 1470.122234] LNet: Skipped 3 previous similar messages [ 1471.060147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1481.732422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1481.760471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1481.797981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1490.129800] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 01:01:23 (1787461283) [ 1490.760150] LNet: 71328:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1490.764972] LNetError: 71328:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1490.769251] LNetError: 71328:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1490.774077] LNet: Removed LNI 192.168.202.55@tcp [ 1490.788606] LNet: Removed LNI 192.168.202.55@tcp1 [ 1491.586288] Key type .llcrypt unregistered [ 1491.592532] Key type ._llcrypt unregistered [ 1499.169831] Key type ._llcrypt registered [ 1499.170920] Key type .llcrypt registered [ 1499.398825] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1499.435386] alg: No test for adler32 (adler32-zlib) [ 1500.414032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1500.455636] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 1511.575970] Lustre: DEBUG MARKER: Initial discovery [ 1511.684730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.155@tcp [ 1514.466952] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1514.818622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1514.892583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.155@tcp [ 1514.941227] LNet: There was an unexpected network error while writing to 192.168.202.155: rc = -22 [ 1514.977241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1515.040656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.202.155@tcp -r 1 -m GET -l 3 [ 1515.081000] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.202.155@tcp -r 1 -m GET -e local_error [ 1515.116992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.202.155@tcp -r 1 -m PUT -l 6 [ 1515.807536] LNet: 1 local NIs in recovery (showing 1): 192.168.202.55@tcp [ 1517.038851] Lustre: DEBUG MARKER: Force oleg255-client.virtnet to discover 192.168.202.155@tcp (in background) [ 1517.143145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.155@tcp [ 1518.713756] Lustre: DEBUG MARKER: Force oleg255-server to discover 192.168.202.55@tcp [ 1520.223902] LNet: There was an unexpected network error while writing to 192.168.202.155: rc = -22 [ 1522.607544] Lustre: DEBUG MARKER: Wait for 72087 [ 1523.622399] LNet: There was an unexpected network error while writing to 192.168.202.155: rc = -22 [ 1524.931781] Lustre: DEBUG MARKER: Finished wait on 72087 [ 1525.090932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1525.166683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1525.447539] LNet: 72549:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1525.452824] LNetError: 72549:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1526.505843] LNet: Removed LNI 192.168.202.55@tcp [ 1527.221747] Key type .llcrypt unregistered [ 1527.225682] Key type ._llcrypt unregistered [ 1547.024915] Key type ._llcrypt registered [ 1547.026480] Key type .llcrypt registered [ 1548.771678] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 01:02:22 (1787461342) [ 1549.647641] Key type .llcrypt unregistered [ 1549.650254] Key type ._llcrypt unregistered [ 1556.912683] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1557.125153] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1557.138461] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1557.147728] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1558.716414] Key type ._llcrypt registered [ 1558.730602] Key type .llcrypt registered [ 1558.939178] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1558.961869] alg: No test for adler32 (adler32-zlib) [ 1559.957075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1560.003626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1560.035058] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 1560.814345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1560.833225] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1568.991728] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 01:02:42 (1787461362) [ 1569.251433] LNet: 74621:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1569.268878] LNetError: 74621:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1569.292131] LNet: Removed LNI 10.1.2.3@tcp [ 1569.321281] LNet: Removed LNI 192.168.202.55@tcp [ 1569.892426] Key type .llcrypt unregistered [ 1569.896420] Key type ._llcrypt unregistered [ 1576.946404] Key type ._llcrypt registered [ 1576.949410] Key type .llcrypt registered [ 1577.161905] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1577.176267] alg: No test for adler32 (adler32-zlib) [ 1578.127485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1578.165704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1578.182241] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 1578.773418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1578.800188] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1579.806272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.202.55@tcp [ 1579.858849] LNetError: Unexpected error -22 connecting to 192.168.202.55@tcp at host 192.168.202.55:988 [ 1585.942241] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 01:02:59 (1787461379) [ 1586.130598] LNet: 75755:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1586.137536] LNetError: 75755:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1586.143191] LNet: Removed LNI 10.1.2.3@tcp [ 1586.183342] LNet: Removed LNI 192.168.202.55@tcp [ 1586.868470] Key type .llcrypt unregistered [ 1586.871365] Key type ._llcrypt unregistered [ 1594.199564] Key type ._llcrypt registered [ 1594.207157] Key type .llcrypt registered [ 1594.511791] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1594.537827] alg: No test for adler32 (adler32-zlib) [ 1595.520338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1595.601806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1595.670267] LNet: Added LNI 192.168.202.55@tcp1 [8/256/0/180] [ 1596.602561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1596.621344] LNet: Added LNI 192.168.202.55@tcp2 [8/256/0/180] [ 1597.833074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.202.55@tcp1 --nid 192.168.202.55@tcp2 [ 1603.780877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.55@tcp1 192.168.202.55@tcp2 [ 1603.815930] LNetError: 76598:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.202.55@tcp1 to 192.168.202.55@tcp2 - Route table may be misconfigured [ 1603.843035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.55@tcp2 192.168.202.55@tcp1 [ 1610.609760] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 01:03:24 (1787461404) [ 1610.910985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1610.999519] LNet: 77171:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1611.009752] LNetError: 77171:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1612.063339] LNet: Removed LNI 192.168.202.55@tcp1 [ 1612.067265] LNetError: 77171:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1612.098327] LNet: Removed LNI 192.168.202.55@tcp2 [ 1612.188538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1612.294671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1612.358757] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 1613.116358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1613.143273] LNet: Added LNI 192.168.202.55@tcp1 [8/256/0/180] [ 1614.210749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.55@tcp [ 1614.269162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.55@tcp 192.168.202.55@tcp [ 1614.315583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.55@tcp1 192.168.202.55@tcp1 [ 1614.358126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.55@tcp -d 192.168.202.55@tcp -e network_timeout [ 1614.445345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.55@tcp -d 192.168.202.55@tcp1 -e network_timeout [ 1614.511912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.55@tcp1 -d 192.168.202.55@tcp -e network_timeout [ 1614.576733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.55@tcp1 -d 192.168.202.55@tcp1 -e network_timeout [ 1614.623361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.55@tcp [ 1614.663109] LNet: There was an unexpected network error while writing to 192.168.202.55: rc = -22 [ 1614.689155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1614.787964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1614.897800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1615.327833] LNet: 1 local NIs in recovery (showing 1): 192.168.202.55@tcp [ 1622.174758] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 01:03:36 (1787461416) [ 1622.282956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1622.323642] LNet: 78024:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1622.338424] LNetError: 78024:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1622.344798] LNet: Removed LNI 192.168.202.55@tcp [ 1623.402468] LNet: Removed LNI 192.168.202.55@tcp1 [ 1623.467372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1623.650446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1624.042623] LNet: 78074:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1624.491547] Key type .llcrypt unregistered [ 1624.496117] Key type ._llcrypt unregistered [ 1637.474543] Key type ._llcrypt registered [ 1637.476575] Key type .llcrypt registered [ 1638.724433] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 01:03:52 (1787461432) [ 1639.243932] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1639.264941] alg: No test for adler32 (adler32-zlib) [ 1640.319042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1642.034677] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1644.409607] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 01:03:57 (1787461437) [ 1644.488951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1644.518233] LNet: 79166:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1644.540979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1644.612721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1644.667867] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 1645.285510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1645.309831] LNet: Added LNI 192.168.202.55@tcp1 [8/256/0/180] [ 1646.294801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.55@tcp [ 1646.336393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.55@tcp1 [ 1646.380461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.55@tcp1 [ 1652.566560] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 01:04:06 (1787461446) [ 1654.665631] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg255-server [ 1656.628438] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 01:04:10 (1787461450) [ 1658.365468] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg255-server [ 1659.945343] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 01:04:13 (1787461453) [ 1661.605424] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg255-server [ 1663.760127] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 01:04:17 (1787461457) [ 1665.436276] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg255-server [ 1667.499310] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 01:04:21 (1787461461) [ 1669.540756] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg255-server [ 1671.629681] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 01:04:25 (1787461465) [ 1673.490162] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg255-server [ 1675.586346] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 01:04:29 (1787461469) [ 1677.744300] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg255-server [ 1679.731185] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 01:04:33 (1787461473) [ 1681.573635] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg255-server [ 1684.225510] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 01:04:37 (1787461477) [ 1687.129190] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg255-server [ 1689.346200] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 01:04:42 (1787461482) [ 1689.454671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1689.486858] LNet: 83045:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1689.492044] LNetError: 83045:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1689.495872] LNet: Removed LNI 192.168.202.55@tcp [ 1689.512071] LNet: Removed LNI 192.168.202.55@tcp1 [ 1689.543453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1689.582485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1689.603728] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 1690.325800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1690.442815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.55@tcp [ 1690.624941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1690.654346] LNet: 83200:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1690.666946] LNetError: 83200:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1690.677140] LNetError: 83200:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1690.696902] LNet: Removed LNI 192.168.202.55@tcp [ 1690.715111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1690.829772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1690.897058] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 1691.826291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.55@tcp [ 1693.204235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1693.259382] LNet: 83355:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1693.266179] LNetError: 83355:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1694.310070] LNet: Removed LNI 192.168.202.55@tcp [ 1694.369286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1694.506913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1694.551289] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 1695.708943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1695.871402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.55@tcp [ 1696.094159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1696.120777] LNetError: 83502:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1697.197726] LNet: Removed LNI 192.168.202.55@tcp [ 1697.269759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1697.353229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1697.384620] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 1698.073330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.55@tcp [ 1698.223573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1698.239332] LNet: 83647:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1698.244228] LNet: 83647:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1699.304907] LNet: Removed LNI 192.168.202.55@tcp [ 1699.335361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1699.451931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1699.479874] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 1699.980144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1700.042827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.55@tcp [ 1700.172783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1700.187900] LNetError: 83794:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1700.193276] LNetError: 83794:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1701.223057] LNet: Removed LNI 192.168.202.55@tcp [ 1701.257405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1701.307577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1701.383465] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 1702.320339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.55@tcp [ 1702.607545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1703.655979] LNet: Removed LNI 192.168.202.55@tcp [ 1703.703384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1703.833687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1703.928513] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 1704.950409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1705.011897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.55@tcp [ 1705.223521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1706.287614] LNet: Removed LNI 192.168.202.55@tcp [ 1706.391621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1706.456977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1706.474900] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 1707.383330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.55@tcp [ 1708.935444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1708.957878] LNet: 84242:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1708.964507] LNet: 84242:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1708.976972] LNetError: 84242:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1708.984232] LNetError: 84242:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1710.063332] LNet: Removed LNI 192.168.202.55@tcp [ 1710.132902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1710.280676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1710.327965] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 1711.834719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1712.015983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.55@tcp [ 1712.496918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1713.574738] LNet: Removed LNI 192.168.202.55@tcp [ 1713.647364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1713.857987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1713.934500] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 1715.363958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.55@tcp [ 1715.683187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1716.779603] LNet: Removed LNI 192.168.202.55@tcp [ 1716.798943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1716.939333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1717.003092] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 1718.052788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1718.173993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.55@tcp [ 1718.514187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1719.603096] LNet: Removed LNI 192.168.202.55@tcp [ 1719.673144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1719.717963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1719.768464] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 1721.058584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.55@tcp [ 1722.537789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1723.632102] LNet: Removed LNI 192.168.202.55@tcp [ 1723.685934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1723.772409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1723.821343] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 1724.870454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1724.951364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1724.997237] LNet: 84970:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1725.001229] LNet: 84970:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1725.009520] LNetError: 84970:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1725.012642] LNetError: 84970:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1725.024829] LNet: Removed LNI 192.168.202.55@tcp [ 1725.047297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1725.103669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1725.150602] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 1726.127771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1726.232302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1726.276848] LNet: Removed LNI 192.168.202.55@tcp [ 1726.303652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1726.369888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1726.492679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1726.545830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1726.623275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1726.807871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1726.905875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1726.992638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1727.080541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1727.124822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1727.176193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1727.223948] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 1728.089452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1728.144475] LNet: Removed LNI 192.168.202.55@tcp [ 1728.160552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1728.260907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1737.361744] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 01:05:30 (1787461530) [ 1737.575740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1737.685535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1737.871206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1737.900099] LNet: Added LNI 192.168.202.55@tcp231 [8/256/0/180] [ 1740.616821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1740.698983] LNet: Removed LNI 192.168.202.55@tcp231 [ 1740.781705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1740.952996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1741.014269] LNet: Added LNI 192.168.202.55@tcp231 [8/256/0/180] [ 1741.025915] LNet: 86007:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1741.631990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1741.692068] LNet: Removed LNI 192.168.202.55@tcp231 [ 1741.732453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1741.798492] LNet: Added LNI 192.168.202.55@tcp231 [8/256/0/180] [ 1743.803615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1743.847774] LNet: Removed LNI 192.168.202.55@tcp231 [ 1743.873602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1744.051474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1744.124856] LNet: Added LNI 192.168.202.55@tcp231 [8/256/0/0] [ 1744.140968] LNet: 86160:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1745.214187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1745.255611] LNet: Removed LNI 192.168.202.55@tcp231 [ 1745.292395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1745.337105] LNet: Added LNI 192.168.202.55@tcp231 [8/256/0/0] [ 1746.832562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1746.874669] LNet: Removed LNI 192.168.202.55@tcp231 [ 1746.911808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1747.087720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1747.149580] LNet: Added LNI 192.168.202.55@tcp231 [8/256/0/60] [ 1747.166336] LNet: 86312:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1747.753907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1747.799437] LNet: Removed LNI 192.168.202.55@tcp231 [ 1747.812783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1747.858989] LNet: Added LNI 192.168.202.55@tcp231 [8/256/0/60] [ 1749.405363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1749.478257] LNet: Removed LNI 192.168.202.55@tcp231 [ 1749.512663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1749.625029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1749.687830] LNet: Added LNI 192.168.202.55@tcp231 [8/256/0/180] [ 1749.696326] LNet: 86463:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1750.409178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1750.469435] LNet: Removed LNI 192.168.202.55@tcp231 [ 1750.501160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1750.541389] LNet: Added LNI 192.168.202.55@tcp231 [8/256/0/180] [ 1759.817361] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 01:05:53 (1787461553) [ 1759.924773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1759.963470] LNet: 87169:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1759.973326] LNet: 87169:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 10 previous similar messages [ 1759.988552] LNetError: 87169:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1760.002188] LNetError: 87169:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1760.020949] LNet: Removed LNI 192.168.202.55@tcp231 [ 1760.065868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1760.151881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1760.189609] LNet: Added LNI 192.168.202.55@tcp232 [8/256/0/180] [ 1760.891349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1760.921982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1760.949434] LNet: Removed LNI 192.168.202.55@tcp232 [ 1760.978361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1761.009541] LNet: Added LNI 192.168.202.55@tcp232 [8/256/0/180] [ 1762.025489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1762.090874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1762.155978] LNet: Removed LNI 192.168.202.55@tcp232 [ 1762.187195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1762.226283] LNet: Added LNI 192.168.202.55@tcp232 [8/256/0/180] [ 1763.363227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1763.396755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1763.425947] LNet: Removed LNI 192.168.202.55@tcp232 [ 1763.443161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1763.470469] LNet: Added LNI 192.168.202.55@tcp232 [8/256/0/180] [ 1764.390916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1764.441623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1764.477903] LNet: Removed LNI 192.168.202.55@tcp232 [ 1764.500834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1772.368061] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 01:06:06 (1787461566) [ 1773.656567] Key type .llcrypt unregistered [ 1773.658418] Key type ._llcrypt unregistered [ 1780.963378] Key type ._llcrypt registered [ 1780.969175] Key type .llcrypt registered [ 1781.143583] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1781.192731] alg: No test for adler32 (adler32-zlib) [ 1782.190463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1782.232333] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 1789.970837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.155@tcp [ 1799.893121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1799.946666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1799.972654] LNet: Added LNI 192.168.202.55@tcp1 [8/256/0/180] [ 1801.448951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.155@tcp -m GET -e local_interrupt -r 2 [ 1801.482263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.155@tcp1 -m GET -e local_interrupt -r 2 [ 1801.524974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp -d 192.168.202.55@tcp -m GET -e local_interrupt -r 2 [ 1801.546285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.55@tcp1 -m GET -e local_interrupt -r 2 [ 1801.582155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp1 -d 192.168.202.155@tcp -m GET -e local_interrupt -r 2 [ 1801.634165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp1 -d 192.168.202.155@tcp1 -m GET -e local_interrupt -r 2 [ 1801.678532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp1 -d 192.168.202.55@tcp -m GET -e local_interrupt -r 2 [ 1801.731257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp1 -d 192.168.202.55@tcp1 -m GET -e local_interrupt -r 2 [ 1801.765949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.55@tcp 192.168.202.155@tcp [ 1801.811393] LNet: There was an unexpected network error while writing to 192.168.202.155: rc = -22 [ 1801.816442] LNet: 1 local NIs in recovery (showing 1): 192.168.202.55@tcp [ 1802.069468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1802.105471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1802.143245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1802.179204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1802.204907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1802.232313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1802.262271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1802.304989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1802.327132] LNet: 88969:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1802.345339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1802.385438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1802.449993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1802.505925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1802.554960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1802.601571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1803.130390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.155@tcp -m GET -e local_dropped -r 2 [ 1803.159847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.155@tcp1 -m GET -e local_dropped -r 2 [ 1803.189785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.55@tcp -m GET -e local_dropped -r 2 [ 1803.225237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.55@tcp1 -m GET -e local_dropped -r 2 [ 1803.269797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp1 -d 192.168.202.155@tcp -m GET -e local_dropped -r 2 [ 1803.305334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp1 -d 192.168.202.155@tcp1 -m GET -e local_dropped -r 2 [ 1803.350882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp1 -d 192.168.202.55@tcp -m GET -e local_dropped -r 2 [ 1803.394017] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp1 -d 192.168.202.55@tcp1 -m GET -e local_dropped -r 2 [ 1803.426556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.55@tcp 192.168.202.155@tcp [ 1803.441603] LNet: There was an unexpected network error while writing to 192.168.202.155: rc = -22 [ 1803.709137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1803.753909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1803.785246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1803.809431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1803.835195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1803.862639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1803.890257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1803.920907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1803.937864] LNet: 89090:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1803.961362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1803.985309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1804.014885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1804.031435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1804.062732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1804.113886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1804.641308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp -d 192.168.202.155@tcp -m GET -e local_aborted -r 2 [ 1804.664438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.155@tcp1 -m GET -e local_aborted -r 2 [ 1804.694703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.55@tcp -m GET -e local_aborted -r 2 [ 1804.734747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp -d 192.168.202.55@tcp1 -m GET -e local_aborted -r 2 [ 1804.773600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp1 -d 192.168.202.155@tcp -m GET -e local_aborted -r 2 [ 1804.817333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp1 -d 192.168.202.155@tcp1 -m GET -e local_aborted -r 2 [ 1804.867882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp1 -d 192.168.202.55@tcp -m GET -e local_aborted -r 2 [ 1804.928308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp1 -d 192.168.202.55@tcp1 -m GET -e local_aborted -r 2 [ 1804.974029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.55@tcp 192.168.202.155@tcp [ 1804.983503] LNet: There was an unexpected network error while writing to 192.168.202.155: rc = -22 [ 1805.258087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1805.326108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1805.355711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1805.395926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1805.441978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1805.509865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1805.543404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1805.574369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1805.596258] LNet: 89210:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1805.620324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1805.658956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1805.696868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1805.752820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1805.834244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1805.891798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1806.503883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp -d 192.168.202.155@tcp -m GET -e local_no_route -r 2 [ 1806.561801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp -d 192.168.202.155@tcp1 -m GET -e local_no_route -r 2 [ 1806.625992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.55@tcp -m GET -e local_no_route -r 2 [ 1806.718625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.55@tcp1 -m GET -e local_no_route -r 2 [ 1806.814149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp1 -d 192.168.202.155@tcp -m GET -e local_no_route -r 2 [ 1806.880445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp1 -d 192.168.202.155@tcp1 -m GET -e local_no_route -r 2 [ 1806.963415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp1 -d 192.168.202.55@tcp -m GET -e local_no_route -r 2 [ 1807.035388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp1 -d 192.168.202.55@tcp1 -m GET -e local_no_route -r 2 [ 1807.100332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.55@tcp 192.168.202.155@tcp [ 1807.133690] LNet: There was an unexpected network error while writing to 192.168.202.155: rc = -22 [ 1807.420930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1807.445348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1807.476858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1807.509581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1807.528559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1807.559468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1807.608721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1807.653027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1807.694300] LNet: 89330:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1807.716973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1807.742821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1807.764597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1807.794497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1807.821880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1807.850780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1808.474039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.155@tcp -m GET -e local_timeout -r 2 [ 1808.520315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp -d 192.168.202.155@tcp1 -m GET -e local_timeout -r 2 [ 1808.587452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp -d 192.168.202.55@tcp -m GET -e local_timeout -r 2 [ 1808.651384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.55@tcp1 -m GET -e local_timeout -r 2 [ 1808.705440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp1 -d 192.168.202.155@tcp -m GET -e local_timeout -r 2 [ 1808.747980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp1 -d 192.168.202.155@tcp1 -m GET -e local_timeout -r 2 [ 1808.792669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp1 -d 192.168.202.55@tcp -m GET -e local_timeout -r 2 [ 1808.831220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp1 -d 192.168.202.55@tcp1 -m GET -e local_timeout -r 2 [ 1808.864583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.55@tcp 192.168.202.155@tcp [ 1809.157196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1809.186722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1809.214403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1809.244258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1809.282310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1809.323975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1809.372076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1809.439897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1809.481141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1809.513713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1809.535598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1809.571457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1809.602215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1809.657657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1810.316895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp -d 192.168.202.155@tcp -m GET -e remote_dropped -r 2 [ 1810.394820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.155@tcp1 -m GET -e remote_dropped -r 2 [ 1810.450558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.55@tcp -m GET -e remote_dropped -r 2 [ 1810.565955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.55@tcp1 -m GET -e remote_dropped -r 2 [ 1810.627493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp1 -d 192.168.202.155@tcp -m GET -e remote_dropped -r 2 [ 1810.668752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp1 -d 192.168.202.155@tcp1 -m GET -e remote_dropped -r 2 [ 1810.701030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp1 -d 192.168.202.55@tcp -m GET -e remote_dropped -r 2 [ 1810.733178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp1 -d 192.168.202.55@tcp1 -m GET -e remote_dropped -r 2 [ 1810.768753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.55@tcp 192.168.202.155@tcp [ 1810.788240] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.155@tcp [ 1811.073120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1811.113774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1811.141314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1811.182051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1811.219211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1811.277994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1811.307731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1811.362566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1811.409079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1811.449222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1811.475426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1811.511511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1811.538778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1811.569302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1817.364866] LNet: 89649:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1817.370304] LNetError: 89649:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1817.378048] LNet: Removed LNI 192.168.202.55@tcp [ 1817.400814] LNet: Removed LNI 192.168.202.55@tcp1 [ 1818.007467] Key type .llcrypt unregistered [ 1818.011544] Key type ._llcrypt unregistered [ 1836.436838] Key type ._llcrypt registered [ 1836.439641] Key type .llcrypt registered [ 1837.848458] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 01:07:11 (1787461631) [ 1838.283750] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1838.294910] alg: No test for adler32 (adler32-zlib) [ 1839.199925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1839.300870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1839.358755] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 1840.463724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1840.510832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.55@tcp [ 1840.537553] LNetError: 90631:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.202.55@tcp to recovery [ 1840.632820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.202.55@tcp [ 1840.780842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1847.885416] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 01:07:21 (1787461641) [ 1848.153923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1848.187347] LNet: 91220:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1848.194851] LNetError: 91220:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1848.211844] LNet: Removed LNI 192.168.202.55@tcp [ 1848.235153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1848.286809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1848.315768] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 1849.098088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1849.120787] LNet: Added LNI 192.168.202.55@tcp1 [8/256/0/180] [ 1849.920226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.202.55@tcp [ 1849.937440] LNetError: 91487:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.202.55@tcp to recovery [ 1850.336088] LNet: 1 local NIs in recovery (showing 1): 192.168.202.55@tcp [ 1850.343388] LNet: There was an unexpected network error while writing to 192.168.202.55: rc = -22 [ 1851.202203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.202.55@tcp [ 1859.429315] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 01:07:33 (1787461653) [ 1861.150847] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg255-server [ 1862.883836] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 01:07:36 (1787461656) [ 1864.587523] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg255-server [ 1866.084456] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 01:07:40 (1787461660) [ 1868.010378] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg255-server [ 1869.793458] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 01:07:43 (1787461663) [ 1869.991754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1870.022551] LNet: 93117:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1870.029448] LNetError: 93117:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1871.071551] LNet: Removed LNI 192.168.202.55@tcp [ 1871.079651] LNetError: 93117:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1871.105347] LNet: Removed LNI 192.168.202.55@tcp1 [ 1871.165131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1871.268497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1871.314969] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 1872.189518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1872.271337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1872.318685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1872.433636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1872.485510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1872.585244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1878.908535] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 01:07:52 (1787461672) [ 1879.287040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1879.325227] LNet: 93849:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1879.343440] LNetError: 93849:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1879.369522] LNet: Removed LNI 192.168.202.55@tcp [ 1879.385958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1879.464069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1879.486566] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 1886.495603] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 01:08:00 (1787461680) [ 1886.754663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1886.804647] LNet: 94551:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1886.848896] LNetError: 94551:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1886.871296] LNet: Removed LNI 192.168.202.55@tcp [ 1886.900963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1886.957607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1886.978527] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 1894.959787] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 01:08:08 (1787461688) [ 1896.322927] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1897.905345] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 01:08:11 (1787461691) [ 1898.522742] LNet: 95586:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1898.528320] LNetError: 95586:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1898.541990] LNet: Removed LNI 192.168.202.55@tcp [ 1899.044697] Key type .llcrypt unregistered [ 1899.049170] Key type ._llcrypt unregistered [ 1906.334338] Key type ._llcrypt registered [ 1906.340053] Key type .llcrypt registered [ 1906.664470] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1906.693919] alg: No test for adler32 (adler32-zlib) [ 1907.635760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1907.669532] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 1917.806322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.155@tcp [ 1924.543533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.202.155@tcp [ 1924.901219] LNet: 96012:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1924.907673] LNetError: 96012:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1924.929244] LNet: Removed LNI 192.168.202.55@tcp [ 1925.317613] Key type .llcrypt unregistered [ 1925.319174] Key type ._llcrypt unregistered [ 1938.315762] Key type ._llcrypt registered [ 1938.320481] Key type .llcrypt registered [ 1939.817544] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1941.537820] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1943.291551] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 01:08:56 (1787461736) [ 1944.804355] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1946.805602] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 01:09:00 (1787461740) [ 1949.108932] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg255-server [ 1951.595828] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 01:09:04 (1787461744) [ 1953.934477] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg255-server [ 1956.812071] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 01:09:09 (1787461749) [ 1959.021795] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1959.120494] alg: No test for adler32 (adler32-zlib) [ 1960.300537] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [ 1960.631083] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 1962.439790] Key type lgssc registered [ 1964.737394] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1988.393763] Key type lgssc unregistered [ 1988.685192] LNet: 99194:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1988.704075] LNetError: 99194:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1988.721777] LNet: Removed LNI 192.168.202.55@tcp [ 1989.304548] Key type .llcrypt unregistered [ 1989.310904] Key type ._llcrypt unregistered [ 2005.901379] Key type ._llcrypt registered [ 2005.903878] Key type .llcrypt registered [ 2007.165428] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 01:10:01 (1787461801) [ 2007.874296] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2007.917029] alg: No test for adler32 (adler32-zlib) [ 2008.856934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2008.989323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2009.102478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2009.244335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2009.303762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2016.468250] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 01:10:10 (1787461810) [ 2016.564852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2016.597754] LNet: 100604:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2016.627433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2016.747463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2016.802028] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 2017.888273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2017.910594] LNet: 100754:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2017.916651] LNetError: 100754:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2017.935781] LNet: Removed LNI 192.168.202.55@tcp [ 2017.957461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2018.012522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2018.041644] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 2018.684459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2018.715093] LNetError: 100782:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2018.731464] LNet: Removed LNI 192.168.202.55@tcp [ 2018.751245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2018.796670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2018.828242] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 2027.844404] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 01:10:21 (1787461821) [ 2028.163636] LNet: 101373:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2028.172880] LNet: 101373:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2028.176571] LNetError: 101373:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2028.192495] LNet: Removed LNI 192.168.202.55@tcp [ 2028.873652] Key type .llcrypt unregistered [ 2028.875364] Key type ._llcrypt unregistered [ 2036.008370] Key type ._llcrypt registered [ 2036.012202] Key type .llcrypt registered [ 2036.268160] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2036.292779] alg: No test for adler32 (adler32-zlib) [ 2037.250411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2037.355743] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 2037.358255] LNet: *** cfs_fail_loc=e003, val=100*** [ 2038.269051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.55@tcp [ 2038.286341] LNet: *** cfs_fail_loc=e003, val=100*** [ 2038.289475] LNet: Skipped 2 previous similar messages [ 2038.301824] Ping reply received with mapping: [ 2038.307148] found metadata reply NI[192.168.202.55@tcp] [ 2038.311501] 37 ca a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2038.316304] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2038.320416] 5a 5a 5a 5a 5a 5a 5a 5a [ 2038.549979] LNet: 101774:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2038.566252] LNetError: 101774:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2038.592866] LNet: Removed LNI 192.168.202.55@tcp [ 2039.372617] Key type .llcrypt unregistered [ 2039.376273] Key type ._llcrypt unregistered [ 2053.631617] Key type ._llcrypt registered [ 2053.633304] Key type .llcrypt registered [ 2055.125123] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 01:10:48 (1787461848) [ 2055.828587] Key type .llcrypt unregistered [ 2055.835509] Key type ._llcrypt unregistered [ 2065.022967] Key type ._llcrypt registered [ 2065.025354] Key type .llcrypt registered [ 2065.263739] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2065.283470] alg: No test for adler32 (adler32-zlib) [ 2066.393138] LNetError: 102833:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2066.933849] Key type .llcrypt unregistered [ 2066.936936] Key type ._llcrypt unregistered [ 2072.035136] Key type ._llcrypt registered [ 2072.038890] Key type .llcrypt registered [ 2073.473121] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 01:11:07 (1787461867) [ 2074.142750] Key type .llcrypt unregistered [ 2074.146459] Key type ._llcrypt unregistered [ 2082.085991] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2082.100409] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2084.962516] Key type ._llcrypt registered [ 2084.964804] Key type .llcrypt registered [ 2085.223533] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2085.234633] alg: No test for adler32 (adler32-zlib) [ 2086.270796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2086.356294] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2099.475424] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 01:11:32 (1787461892) [ 2099.656905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2099.705406] LNet: 104638:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2099.730417] LNetError: 104638:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2099.750667] LNet: Removed LNI 10.1.2.3@tcp [ 2099.776642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2100.821765] Key type .llcrypt unregistered [ 2100.823460] Key type ._llcrypt unregistered [ 2109.024607] Key type ._llcrypt registered [ 2109.042470] Key type .llcrypt registered [ 2109.323907] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2109.377512] alg: No test for adler32 (adler32-zlib) [ 2110.529599] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 2110.661808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.55@tcp [ 2110.922417] LNet: 104931:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2110.927676] LNetError: 104931:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2110.955443] LNet: Removed LNI 192.168.202.55@tcp [ 2111.718393] Key type .llcrypt unregistered [ 2111.720673] Key type ._llcrypt unregistered [ 2127.952128] Key type ._llcrypt registered [ 2127.956514] Key type .llcrypt registered [ 2129.609970] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 01:12:02 (1787461922) [ 2130.488330] Key type .llcrypt unregistered [ 2130.493407] Key type ._llcrypt unregistered [ 2135.679594] Key type ._llcrypt registered [ 2135.681266] Key type .llcrypt registered [ 2136.052599] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2136.086297] alg: No test for adler32 (adler32-zlib) [ 2144.182588] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 01:12:17 (1787461937) [ 2144.486086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2144.531671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2144.594537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2144.647191] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 2145.559631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2145.593817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2145.652826] LNet: Removed LNI 192.168.202.55@tcp [ 2146.084377] LNet: 106688:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2146.663682] Key type .llcrypt unregistered [ 2146.670989] Key type ._llcrypt unregistered [ 2161.455549] Key type ._llcrypt registered [ 2161.457953] Key type .llcrypt registered [ 2162.953913] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 01:12:36 (1787461956) [ 2164.024195] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2164.043175] alg: No test for adler32 (adler32-zlib) [ 2164.996975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2165.046257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2165.063062] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 2165.852533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.55@tcp [ 2165.909759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.202.55@tcp [ 2165.935372] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2165.942671] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2165.945856] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2165, 0, 0 [ 2165.951545] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2165.953974] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2165.957188] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2165.960187] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2165, 0, 0 [ 2165.964042] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2165.965856] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2165.968665] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2165.971433] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2165, 0, 2215 [ 2165.975707] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2165.977664] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2165.981317] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2165.985357] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2165, 0, 2215 [ 2165.990287] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2173.711237] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 01:12:47 (1787461967) [ 2174.256532] LNet: 108229:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2174.262320] LNetError: 108229:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2175.360377] LNet: Removed LNI 192.168.202.55@tcp [ 2176.362158] Key type .llcrypt unregistered [ 2176.367098] Key type ._llcrypt unregistered [ 2183.902721] Key type ._llcrypt registered [ 2183.906071] Key type .llcrypt registered [ 2184.327777] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2184.395486] alg: No test for adler32 (adler32-zlib) [ 2185.472947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2185.500034] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 2195.183828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.155@tcp [ 2206.909214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2206.934219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2206.952478] LNet: Added LNI 192.168.202.55@tcp1 [8/256/0/180] [ 2207.713072] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2207.847273] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2207.855893] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2207.865877] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2210.346187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2210.399239] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2221.132564] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 01:13:35 (1787462015) [ 2221.226061] LNet: 109734:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2221.237162] LNetError: 109734:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2222.239147] LNet: Removed LNI 192.168.202.55@tcp [ 2222.241246] LNetError: 109734:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2222.250407] LNet: Removed LNI 192.168.202.55@tcp1 [ 2222.261057] LNet: Removed LNI 10.1.2.3@tcp99 [ 2223.027714] Key type .llcrypt unregistered [ 2223.030049] Key type ._llcrypt unregistered [ 2231.669090] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2231.686764] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2234.590340] Key type ._llcrypt registered [ 2234.593342] Key type .llcrypt registered [ 2234.929686] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2234.989815] alg: No test for adler32 (adler32-zlib) [ 2236.052605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2236.148665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2236.199357] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 2237.338560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2237.360569] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2240.812221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2240.906046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2240.981122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2241.030926] LNetError: 110492:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2241.078232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2241.179824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.55@tcp [ 2241.974972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2242.056264] LNet: 110512:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2242.071417] LNetError: 110512:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2242.085390] LNet: Removed LNI 10.1.2.3@tcp [ 2243.112946] LNet: Removed LNI 192.168.202.55@tcp [ 2244.369405] Key type .llcrypt unregistered [ 2244.371131] Key type ._llcrypt unregistered [ 2260.429844] Key type ._llcrypt registered [ 2260.432559] Key type .llcrypt registered [ 2261.849811] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 01:14:15 (1787462055) [ 2262.264855] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2262.277459] alg: No test for adler32 (adler32-zlib) [ 2263.264028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2263.329672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2263.376993] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 2264.247727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.55@tcp [ 2264.340560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg255-client@tcp [ 2272.666124] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 01:14:25 (1787462065) [ 2272.840324] LNet: 112043:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2272.846560] LNetError: 112043:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2272.886512] LNet: Removed LNI 192.168.202.55@tcp [ 2273.762734] Key type .llcrypt unregistered [ 2273.768253] Key type ._llcrypt unregistered [ 2281.664623] Key type ._llcrypt registered [ 2281.666663] Key type .llcrypt registered [ 2281.860895] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2281.871781] alg: No test for adler32 (adler32-zlib) [ 2282.760896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2282.784719] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 2282.815769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2289.627689] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 01:14:43 (1787462083) [ 2291.152897] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2292.970563] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 01:14:46 (1787462086) [ 2294.531693] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2296.669742] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 01:14:50 (1787462090) [ 2296.799894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2296.833382] LNet: 113472:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2296.844321] LNetError: 113472:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2296.865344] LNet: Removed LNI 192.168.202.55@tcp [ 2296.926597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2296.978552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2296.999186] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 2297.670318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2297.721543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2303.930856] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 01:14:58 (1787462098) [ 2304.362856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2304.415987] LNet: 114187:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2304.425810] LNetError: 114187:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2304.446856] LNet: Removed LNI 192.168.202.55@tcp [ 2304.494439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2304.588539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2304.635971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2305.336211] Key type .llcrypt unregistered [ 2305.347239] Key type ._llcrypt unregistered [ 2317.803540] Key type ._llcrypt registered [ 2317.805685] Key type .llcrypt registered [ 2319.316359] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 01:15:12 (1787462112) [ 2319.710242] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2319.719839] alg: No test for adler32 (adler32-zlib) [ 2320.596544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2320.641188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2320.657490] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 2321.232499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2321.289491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.55@tcp [ 2326.954848] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 01:15:20 (1787462120) [ 2327.078395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2327.109662] LNet: 115741:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2327.123240] LNetError: 115741:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2328.166478] LNet: Removed LNI 192.168.202.55@tcp [ 2328.216681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2328.296745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2328.354296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2335.842760] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 01:15:29 (1787462129) [ 2336.494547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2336.570487] LNet: 116327:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2336.606544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2336.686296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2337.249519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2337.273744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2337.335456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2337.674970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2337.699679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2337.787498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2337.809397] LNet: 116360:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2337.843166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2337.929207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2338.499516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2338.531949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2338.633995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2339.348301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2339.390170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2345.883934] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 01:15:39 (1787462139) [ 2345.959400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2345.981101] LNet: 116953:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2346.014221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2346.070105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2346.105643] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 2346.671222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2346.696420] LNet: Added LNI 192.168.202.55@tcp2 [8/256/0/180] [ 2347.359032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2347.414676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.55@tcp [ 2347.616186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2347.637536] LNetError: 117240:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2347.643267] LNet: Removed LNI 192.168.202.55@tcp [ 2347.654291] LNet: Removed LNI 192.168.202.55@tcp2 [ 2347.679193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2347.721353] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 2347.725978] LNet: Added LNI 192.168.202.55@tcp2 [8/256/0/180] [ 2347.730521] LNet: 117246:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2348.322611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.55@tcp [ 2348.551227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2349.601888] LNet: Removed LNI 192.168.202.55@tcp [ 2349.603941] LNetError: 117283:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2349.615968] LNetError: 117283:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2349.646162] LNet: Removed LNI 192.168.202.55@tcp2 [ 2349.732798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2349.796991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2349.841081] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 2350.670844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2350.711837] LNet: Added LNI 192.168.202.55@tcp2 [8/256/0/180] [ 2352.034358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.55@tcp [ 2352.157385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2352.301756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.55@tcp [ 2352.721322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2352.781794] LNet: 117568:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2352.787534] LNet: 117568:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2352.809754] LNetError: 117568:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2352.815771] LNet: Removed LNI 192.168.202.55@tcp [ 2353.836883] LNet: Removed LNI 192.168.202.55@tcp2 [ 2353.877249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2354.010505] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 2354.028375] LNet: Added LNI 192.168.202.55@tcp2 [8/256/0/180] [ 2354.032346] LNet: 117574:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2354.639211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.55@tcp [ 2354.903741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2354.966382] LNet: Removed LNI 192.168.202.55@tcp [ 2356.007896] LNet: Removed LNI 192.168.202.55@tcp2 [ 2356.049103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2356.170996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2356.224327] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 2357.193128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2357.229815] LNet: Added LNI 192.168.202.55@tcp2 [8/256/0/180] [ 2358.314653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2358.369802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.55@tcp [ 2358.690453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2358.721773] LNetError: 117891:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2358.728819] LNetError: 117891:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2358.738723] LNet: Removed LNI 192.168.202.55@tcp [ 2358.748277] LNet: Removed LNI 192.168.202.55@tcp2 [ 2358.799507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2358.913234] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 2358.920746] LNet: Added LNI 192.168.202.55@tcp2 [8/256/0/180] [ 2358.926964] LNet: 117897:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2359.473053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.55@tcp [ 2359.697903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2360.735244] LNet: Removed LNI 192.168.202.55@tcp [ 2360.767649] LNet: Removed LNI 192.168.202.55@tcp2 [ 2360.819455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2360.900727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2360.951869] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 2361.726472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2361.741711] LNet: Added LNI 192.168.202.55@tcp2 [8/256/0/180] [ 2362.565942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.55@tcp [ 2362.606143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2362.682269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.55@tcp [ 2362.865974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2362.884309] LNet: 118219:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2362.889724] LNet: 118219:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2363.935167] LNet: Removed LNI 192.168.202.55@tcp [ 2364.966394] LNet: Removed LNI 192.168.202.55@tcp2 [ 2364.999075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2365.107243] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 2365.113877] LNet: Added LNI 192.168.202.55@tcp2 [8/256/0/180] [ 2365.120462] LNet: 118225:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2365.810819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.55@tcp [ 2366.035752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2366.053851] LNet: Removed LNI 192.168.202.55@tcp [ 2367.085391] LNet: Removed LNI 192.168.202.55@tcp2 [ 2367.168402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2367.243267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2367.301573] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 2368.231393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2368.263869] LNet: Added LNI 192.168.202.55@tcp2 [8/256/0/180] [ 2369.172474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2369.230271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2369.339590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.55@tcp [ 2369.774214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2369.798435] LNetError: 118559:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2369.808889] LNetError: 118559:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2369.822065] LNet: Removed LNI 192.168.202.55@tcp [ 2369.853679] LNet: Removed LNI 192.168.202.55@tcp2 [ 2369.888955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2369.979125] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 2369.988045] LNet: Added LNI 192.168.202.55@tcp2 [8/256/0/180] [ 2369.994211] LNet: 118565:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2370.676717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.55@tcp [ 2370.991890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2372.063564] LNet: Removed LNI 192.168.202.55@tcp [ 2372.086630] LNet: Removed LNI 192.168.202.55@tcp2 [ 2372.154881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2372.256377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2372.315066] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 2373.097480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2373.113369] LNet: Added LNI 192.168.202.55@tcp2 [8/256/0/180] [ 2373.988056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.55@tcp [ 2374.021738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2374.058951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2374.143147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.55@tcp [ 2374.526805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2374.543448] LNet: Removed LNI 192.168.202.55@tcp [ 2375.590049] LNet: Removed LNI 192.168.202.55@tcp2 [ 2375.626376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2375.702392] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 2375.712912] LNet: Added LNI 192.168.202.55@tcp2 [8/256/0/180] [ 2376.373595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.55@tcp [ 2384.109883] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2386.197945] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 01:16:19 (1787462179) [ 2386.385151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2386.426900] LNet: 119680:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2386.439381] LNet: 119680:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2386.463734] LNetError: 119680:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2386.480841] LNetError: 119680:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2386.510626] LNet: Removed LNI 192.168.202.55@tcp [ 2387.558070] LNet: Removed LNI 192.168.202.55@tcp2 [ 2387.617482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2387.959749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2388.018348] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 2388.926478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2388.949696] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2389.977977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2389.999635] LNet: Removed LNI 10.1.2.3@tcp [ 2390.006704] LNet: Removed LNI 192.168.202.55@tcp [ 2390.018895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2390.066872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2390.090881] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 2390.096404] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2390.348192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2390.362268] LNet: Removed LNI 192.168.202.55@tcp [ 2390.370919] LNet: Removed LNI 10.1.2.3@tcp [ 2398.367610] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 01:16:32 (1787462192) [ 2398.992807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2399.092435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2399.197792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2399.258465] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 2400.374442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2400.423926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.55@tcp [ 2400.472656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.55@tcp [ 2400.496898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.55@tcp [ 2400.521791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.55@tcp [ 2400.571177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.55@tcp [ 2400.630552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.55@tcp [ 2400.696607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.55@tcp [ 2400.745384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.55@tcp [ 2400.776732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.55@tcp [ 2400.819561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.55@tcp [ 2400.999468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2408.496120] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 01:16:42 (1787462202) [ 2408.889565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2409.958880] LNet: Removed LNI 192.168.202.55@tcp [ 2410.024403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2410.234431] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2410.401739] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2410.412711] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2410.428605] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2413.037562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2413.080315] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 2413.893891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2413.916793] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2416.881821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.55@tcp [ 2416.961389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2417.071779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.202.55@tcp -m PUT -e local_timeout -r 1 [ 2417.180656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2417.298501] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2417.303922] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2417.544960] LNetError: 121367:0:(lib-move.c:4801:lnet_parse()) 192.168.202.55@tcp, src 192.168.202.55@tcp: Dropping PUT (error -108 looking up sender) [ 2418.600577] LNet: 121829:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2418.611489] LNet: 121829:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2418.626606] LNetError: 121829:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2418.635435] LNetError: 121829:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 2418.648926] LNet: Removed LNI 10.1.2.3@tcp [ 2419.702355] LNet: Removed LNI 192.168.202.55@tcp [ 2420.565425] Key type .llcrypt unregistered [ 2420.567319] Key type ._llcrypt unregistered [ 2434.371816] Key type ._llcrypt registered [ 2434.375166] Key type .llcrypt registered [ 2436.410951] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 01:17:09 (1787462229) [ 2437.154955] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2437.172188] alg: No test for adler32 (adler32-zlib) [ 2438.178651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2438.660619] LNet: 122683:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2439.403664] Key type .llcrypt unregistered [ 2439.408121] Key type ._llcrypt unregistered [ 2446.724721] Key type ._llcrypt registered [ 2446.735980] Key type .llcrypt registered [ 2447.157531] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2447.197770] alg: No test for adler32 (adler32-zlib) [ 2448.203686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2448.272620] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 2456.993640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.155@tcp [ 2457.240843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2457.735560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.155@tcp [ 2457.963838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2457.994432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2458.024348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2458.059694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2458.109404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2458.136859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2458.169902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2458.191764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2458.203927] LNet: 123137:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2458.218341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2458.243643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2458.279778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2458.308756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2458.338253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2458.365848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2458.936901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2459.237099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2459.274897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2459.299592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2459.325843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2459.360242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2459.396772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2459.429262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2459.472472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2459.499920] LNet: 123255:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2459.517266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2459.539247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2459.551285] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.155@tcp [ 2459.571284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2459.592873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2459.622192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2459.672445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2460.447611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.155@tcp [ 2460.751793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2460.793144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2460.835383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2460.865586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2460.903892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2460.942182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2461.017180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2461.066708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2461.101947] LNet: 123372:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2461.118909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2461.154943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2461.192789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2461.228835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2461.278395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2461.310139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2462.013887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2462.435384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2462.467279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2462.493683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2462.525467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2462.582285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2462.621555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2462.645964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2462.683177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2462.729043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2462.761897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2462.793187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2462.817649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2462.853187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2462.887540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2463.183613] LNet: 123539:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2463.190395] LNetError: 123539:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2463.207322] LNet: Removed LNI 192.168.202.55@tcp [ 2463.627299] Key type .llcrypt unregistered [ 2463.628888] Key type ._llcrypt unregistered [ 2486.249523] Key type ._llcrypt registered [ 2486.251662] Key type .llcrypt registered [ 2487.682081] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 01:18:01 (1787462281) [ 2488.455846] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2488.480309] alg: No test for adler32 (adler32-zlib) [ 2489.514472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2490.010523] LNet: 124428:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2490.952419] Key type .llcrypt unregistered [ 2490.953913] Key type ._llcrypt unregistered [ 2499.150034] Key type ._llcrypt registered [ 2499.161142] Key type .llcrypt registered [ 2499.539350] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2499.563701] alg: No test for adler32 (adler32-zlib) [ 2500.430484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2500.448109] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 2507.212347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.155@tcp [ 2517.124744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2517.180903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2517.207254] LNet: Added LNI 192.168.202.55@tcp1 [8/256/0/180] [ 2517.915707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2518.732466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.155@tcp [ 2518.879711] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.155@tcp [ 2519.014723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2519.041510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2519.064809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2519.089890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2519.120602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2519.146399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2519.171756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2519.195717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2519.209368] LNet: 125085:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2519.235566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2519.276212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2519.321952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2519.354251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2519.383929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2519.433060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2520.165500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2520.403668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2520.425644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2520.452985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2520.483581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2520.508505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2520.539646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2520.559280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2520.577570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2520.593883] LNet: 125203:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2520.611726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2520.628535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2520.655522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2520.680401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2520.705276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2520.733434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2521.293775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.155@tcp [ 2521.521712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2521.548639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2521.582350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2521.614355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2521.652835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2521.682729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2521.714319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2521.751722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2521.769968] LNet: 125320:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2521.789717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2521.823732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2521.861760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2521.891981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2521.916619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2521.963748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2522.741091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2523.026930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2523.054426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2523.078644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2523.128400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2523.178661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2523.236558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2523.291838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2523.333587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2523.384314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2523.445448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2523.506278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2523.543741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2523.592748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2523.670464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2528.944517] LNet: 125522:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2528.951258] LNetError: 125522:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2528.964262] LNet: Removed LNI 192.168.202.55@tcp [ 2528.986896] LNet: Removed LNI 192.168.202.55@tcp1 [ 2529.494628] Key type .llcrypt unregistered [ 2529.500540] Key type ._llcrypt unregistered [ 2548.441426] Key type ._llcrypt registered [ 2548.444358] Key type .llcrypt registered [ 2550.112754] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 01:19:03 (1787462343) [ 2550.728238] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2550.747550] alg: No test for adler32 (adler32-zlib) [ 2551.661992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2551.700853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2551.720149] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 2552.364930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2552.404937] LNet: Added LNI 192.168.202.55@tcp2 [8/256/0/180] [ 2553.402307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.55@tcp [ 2553.518489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2554.591316] LNet: Removed LNI 192.168.202.55@tcp [ 2561.242790] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 01:19:15 (1787462355) [ 2563.027474] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2564.465535] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 01:19:18 (1787462358) [ 2571.966786] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 01:19:25 (1787462365) [ 2579.870330] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 01:19:33 (1787462373) [ 2587.701692] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 01:19:41 (1787462381) [ 2596.133726] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 01:19:49 (1787462389) [ 2603.597160] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 01:19:57 (1787462397) [ 2611.403872] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 01:20:04 (1787462404) [ 2620.106499] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 01:20:13 (1787462413) [ 2629.606754] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 01:20:22 (1787462422) [ 2638.259146] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 01:20:31 (1787462431) [ 2645.253791] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 01:20:39 (1787462439) [ 2654.333426] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 01:20:47 (1787462447) [ 2662.520903] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 01:20:56 (1787462456) [ 2668.963752] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 01:21:03 (1787462463) [ 2677.539705] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 01:21:11 (1787462471) [ 2685.710415] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 01:21:19 (1787462479) [ 2693.607338] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 01:21:27 (1787462487) [ 2701.182704] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 01:21:35 (1787462495) [ 2701.791941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2701.890494] LNet: 137221:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2701.894490] LNetError: 137221:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2701.915978] LNet: Removed LNI 192.168.202.55@tcp2 [ 2701.963057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2702.206843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2702.240713] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 2704.008183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2704.059641] LNet: 137363:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2704.071803] LNetError: 137363:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2704.094621] LNet: Removed LNI 192.168.202.55@tcp [ 2704.134926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2704.225081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2704.285297] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 2714.494908] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 01:21:47 (1787462507) [ 2724.828423] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 01:21:57 (1787462517) [ 2733.807915] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 01:22:07 (1787462527) [ 2743.740768] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 01:22:16 (1787462536) [ 2753.908910] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 01:22:27 (1787462547) [ 2763.683162] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 01:22:36 (1787462556) [ 2771.168826] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 01:22:44 (1787462564) [ 2778.812775] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 01:22:52 (1787462572) [ 2786.663912] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 01:23:00 (1787462580) [ 2794.096605] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 01:23:07 (1787462587) [ 2802.008720] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 01:23:15 (1787462595) [ 2813.272686] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 01:23:26 (1787462606) [ 2823.404647] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2573 sec ========== 01:23:36 (1787462616) [ 2823.837021] LNet: 144832:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2823.849616] LNetError: 144832:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2823.867202] LNet: Removed LNI 192.168.202.55@tcp [ 2824.793540] Key type .llcrypt unregistered [ 2824.796364] Key type ._llcrypt unregistered [ 2839.432256] Key type ._llcrypt registered [ 2839.445497] Key type .llcrypt registered [ 2839.750742] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2839.760300] alg: No test for adler32 (adler32-zlib) [ 2841.104835] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [ 2841.505919] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 2843.383704] Key type lgssc registered [ 2844.944503] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2924.490272] Lustre: Mounted lustre-client - version 2.17.57_65_gc740cad [ 2931.117662] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2942.602681] Lustre: Unmounted lustre-client [ 2970.261258] Key type lgssc unregistered [ 2970.573263] LNet: 147796:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2970.579611] LNetError: 147796:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2970.605113] LNet: Removed LNI 192.168.202.55@tcp [ 2971.346909] Key type .llcrypt unregistered [ 2971.348646] Key type ._llcrypt unregistered