[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 423942619 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003262] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005010] kvm-guest: setup PV IPIs [ 0.007835] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008018] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009009] pid_max: default: 32768 minimum: 301 [ 0.010102] LSM: Security Framework initializing [ 0.011034] Yama: becoming mindful. [ 0.012019] SELinux: Initializing. [ 0.013070] *** VALIDATE selinux *** [ 0.021474] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026225] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027166] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028110] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029092] *** VALIDATE tmpfs *** [ 0.030438] *** VALIDATE proc *** [ 0.031253] *** VALIDATE cgroup *** [ 0.032012] *** VALIDATE cgroup2 *** [ 0.033267] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034133] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035022] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036030] Spectre V2 : User space: Vulnerable [ 0.037013] Speculative Store Bypass: Vulnerable [ 0.039822] debug: unmapping init [mem 0xffffffff8a459000-0xffffffff8a460fff] [ 0.042000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042676] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043027] ... version: 2 [ 0.044013] ... bit width: 48 [ 0.045016] ... generic registers: 4 [ 0.046013] ... value mask: 0000ffffffffffff [ 0.047013] ... max period: 00007fffffffffff [ 0.048015] ... fixed-purpose events: 3 [ 0.049014] ... event mask: 000000070000000f [ 0.050281] rcu: Hierarchical SRCU implementation. [ 0.052330] smp: Bringing up secondary CPUs ... [ 0.053467] x86: Booting SMP configuration: [ 0.054020] .... node #0, CPUs: #1 #2 #3 [ 0.057454] smp: Brought up 1 node, 4 CPUs [ 0.059012] smpboot: Max logical packages: 1 [ 0.060016] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.232900] node 0 deferred pages initialised in 170ms [ 0.237023] devtmpfs: initialized [ 0.238227] x86/mm: Memory block size: 128MB [ 0.240507] gcov: version magic: 0x41383552 [ 0.242159] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.243077] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.244267] pinctrl core: initialized pinctrl subsystem [ 0.245211] [ 0.245577] ************************************************************* [ 0.246012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.247013] ** ** [ 0.248008] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.249011] ** ** [ 0.250011] ** This means that this kernel is built to expose internal ** [ 0.251014] ** IOMMU data structures, which may compromise security on ** [ 0.252011] ** your system. ** [ 0.253012] ** ** [ 0.254012] ** If you see this message and you are not debugging the ** [ 0.255012] ** kernel, report this immediately to your vendor! ** [ 0.256009] ** ** [ 0.257011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.258029] ************************************************************* [ 0.259595] NET: Registered protocol family 16 [ 0.260453] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.261060] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.262067] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.263459] cpuidle: using governor menu [ 0.266023] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.269647] PCI: Using configuration type 1 for base access [ 0.271151] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.282057] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.283019] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.285084] cryptd: max_cpu_qlen set to 1000 [ 0.289000] ACPI: Added _OSI(Module Device) [ 0.291023] ACPI: Added _OSI(Processor Device) [ 0.294015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.295015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.301102] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.308091] ACPI: Interpreter enabled [ 0.310106] ACPI: PM: (supports S0 S3 S4 S5) [ 0.311017] ACPI: Using IOAPIC for interrupt routing [ 0.314200] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.317775] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.328704] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.331316] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.335030] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.338106] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.344759] acpiphp: Slot [2] registered [ 0.346187] acpiphp: Slot [5] registered [ 0.348194] acpiphp: Slot [6] registered [ 0.350239] acpiphp: Slot [3] registered [ 0.352149] acpiphp: Slot [4] registered [ 0.353143] acpiphp: Slot [7] registered [ 0.355150] acpiphp: Slot [8] registered [ 0.357186] acpiphp: Slot [9] registered [ 0.359157] acpiphp: Slot [10] registered [ 0.361173] acpiphp: Slot [11] registered [ 0.362143] acpiphp: Slot [12] registered [ 0.364149] acpiphp: Slot [13] registered [ 0.366161] acpiphp: Slot [14] registered [ 0.368145] acpiphp: Slot [15] registered [ 0.369230] acpiphp: Slot [16] registered [ 0.371140] acpiphp: Slot [17] registered [ 0.373176] acpiphp: Slot [18] registered [ 0.375176] acpiphp: Slot [19] registered [ 0.377118] acpiphp: Slot [20] registered [ 0.379161] acpiphp: Slot [21] registered [ 0.381212] acpiphp: Slot [22] registered [ 0.384171] acpiphp: Slot [23] registered [ 0.386153] acpiphp: Slot [24] registered [ 0.387185] acpiphp: Slot [25] registered [ 0.389145] acpiphp: Slot [26] registered [ 0.391151] acpiphp: Slot [27] registered [ 0.393142] acpiphp: Slot [28] registered [ 0.395174] acpiphp: Slot [29] registered [ 0.397225] acpiphp: Slot [30] registered [ 0.399202] acpiphp: Slot [31] registered [ 0.401122] PCI host bridge to bus 0000:00 [ 0.402041] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.405033] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.408041] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.411039] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.414037] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.417035] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.419196] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.421885] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.425204] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.431023] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.436154] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.439025] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.441035] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.444022] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.447095] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.449824] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.453053] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.457011] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.462012] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.473134] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.478017] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.484262] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.494021] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.500020] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.514015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.522506] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.529013] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.535015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.549986] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.559996] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.562427] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.565386] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.568348] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.571229] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.577055] iommu: Default domain type: Passthrough [ 0.579345] SCSI subsystem initialized [ 0.581150] ACPI: bus type USB registered [ 0.582114] usbcore: registered new interface driver usbfs [ 0.584091] usbcore: registered new interface driver hub [ 0.586089] usbcore: registered new device driver usb [ 0.588202] pps_core: LinuxPPS API ver. 1 registered [ 0.590012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.592072] PTP clock support registered [ 0.594143] EDAC MC: Ver: 3.0.0 [ 0.596120] PCI: Using ACPI for IRQ routing [ 0.598529] NetLabel: Initializing [ 0.600013] NetLabel: domain hash size = 128 [ 0.601011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.603086] NetLabel: unlabeled traffic allowed by default [ 0.606080] vgaarb: loaded [ 0.607520] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.610026] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.616011] clocksource: Switched to clocksource kvm-clock [ 0.726960] VFS: Disk quotas dquot_6.6.0 [ 0.729394] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.732838] *** VALIDATE ramfs *** [ 0.734376] *** VALIDATE hugetlbfs *** [ 0.736274] pnp: PnP ACPI init [ 0.738964] pnp: PnP ACPI: found 6 devices [ 0.755871] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.759398] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.761875] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.764286] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.767045] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.769361] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.772373] NET: Registered protocol family 2 [ 0.775209] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.780832] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.785558] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.790750] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.794860] TCP: Hash tables configured (established 65536 bind 65536) [ 0.798619] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.802846] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.806043] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.809502] NET: Registered protocol family 1 [ 0.812836] RPC: Registered named UNIX socket transport module. [ 0.815008] RPC: Registered udp transport module. [ 0.816797] RPC: Registered tcp transport module. [ 0.818419] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.821162] NET: Registered protocol family 44 [ 0.822497] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.824408] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.826168] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.827908] PCI: CLS 0 bytes, default 64 [ 0.829707] Unpacking initramfs... [ 2.236710] debug: unmapping init [mem 0xffff8a5d7cc64000-0xffff8a5d7ffcffff] [ 2.242104] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.244297] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.247238] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.775426] Initialise system trusted keyrings [ 2.777649] Key type blacklist registered [ 2.780046] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.792567] zbud: loaded [ 2.796101] *** VALIDATE nfs *** [ 2.797130] *** VALIDATE nfs4 *** [ 2.798562] pstore: using deflate compression [ 2.801327] Platform Keyring initialized [ 2.907686] NET: Registered protocol family 38 [ 2.909545] Key type asymmetric registered [ 2.910987] Asymmetric key parser 'x509' registered [ 2.912808] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.916494] io scheduler mq-deadline registered [ 2.918287] io scheduler kyber registered [ 2.920056] io scheduler bfq registered [ 2.922186] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.925611] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.930477] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.933790] ACPI: Power Button [PWRF] [ 2.940335] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.947742] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.961373] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.990164] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.022354] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.027802] Non-volatile memory driver v1.3 [ 3.030077] Linux agpgart interface v0.103 [ 3.067399] virtio_blk virtio1: [vda] 146016 512-byte logical blocks (74.8 MB/71.3 MiB) [ 3.070876] vda: detected capacity change from 0 to 74760192 [ 3.086673] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.089152] vdb: detected capacity change from 0 to 1073741824 [ 3.097395] libphy: Fixed MDIO Bus: probed [ 3.102670] usbcore: registered new interface driver usbserial_generic [ 3.104979] usbserial: USB Serial support registered for generic [ 3.108063] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.112563] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.114181] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.116432] mousedev: PS/2 mouse device common for all mice [ 3.119572] rtc_cmos 00:05: RTC can wake from S4 [ 3.122306] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.124344] rtc_cmos 00:05: registered as rtc0 [ 3.128408] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.130357] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.132539] intel_pstate: CPU model not supported [ 3.138579] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.142509] hid: raw HID events driver (C) Jiri Kosina [ 3.144815] usbcore: registered new interface driver usbhid [ 3.146940] usbhid: USB HID core driver [ 3.148771] drop_monitor: Initializing network drop monitor service [ 3.151525] Initializing XFRM netlink socket [ 3.153529] NET: Registered protocol family 10 [ 3.156776] Segment Routing with IPv6 [ 3.158585] NET: Registered protocol family 17 [ 3.160944] mpls_gso: MPLS GSO support [ 3.166828] RAS: Correctable Errors collector initialized. [ 3.169467] AVX version of gcm_enc/dec engaged. [ 3.171541] AES CTR mode by8 optimization enabled [ 3.265546] sched_clock: Marking stable (3265524867, 0)->(4148009173, -882484306) [ 3.269279] registered taskstats version 1 [ 3.270974] Loading compiled-in X.509 certificates [ 3.272811] zswap: loaded using pool lzo/zbud [ 3.294922] Key type big_key registered [ 3.307164] Key type encrypted registered [ 3.308968] ima: No TPM chip found, activating TPM-bypass! [ 3.310660] ima: Allocated hash algorithm: sha1 [ 3.312334] ima: No architecture policies found [ 3.314423] evm: Initialising EVM extended attributes: [ 3.316823] evm: security.selinux [ 3.318155] evm: security.ima [ 3.319120] evm: security.capability [ 3.320491] evm: HMAC attrs: 0x1 [ 3.322934] rtc_cmos 00:05: setting system clock to 2026-08-21 18:59:25 UTC (1787338765) [ 3.329918] debug: unmapping init [mem 0xffffffff8b403000-0xffffffff8b5fffff] [ 3.332968] debug: unmapping init [mem 0xffffffff8a182000-0xffffffff8a458fff] [ 3.342218] Write protecting the kernel read-only data: 28672k [ 3.346036] debug: unmapping init [mem 0xffffffff88803000-0xffffffff889fffff] [ 3.349908] debug: unmapping init [mem 0xffffffff89114000-0xffffffff891fffff] [ 3.395408] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.404332] systemd[1]: Detected virtualization kvm. [ 3.406364] systemd[1]: Detected architecture x86-64. [ 3.408186] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.434503] systemd[1]: No hostname configured. [ 3.436537] systemd[1]: Set hostname to . [ 3.440286] random: systemd: uninitialized urandom read (16 bytes read) [ 3.444667] systemd[1]: Initializing machine ID from random generator. [ 3.503810] random: ln: uninitialized urandom read (6 bytes read) [ 3.582311] random: systemd: uninitialized urandom read (16 bytes read) [ 3.585201] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.592467] systemd[1]: Reached target Local Encrypted Volumes. [ OK ] Reached target Local Encrypted Volumes. [ 3.597385] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Local File Systems. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Initrd Root Device. Starting Setup Virtual Console... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Slices. Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.312619] device-mapper: uevent: version 1.0.3 [ 4.314792] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.096936] virtio_net virtio0 ens2: renamed from eth0 [ 5.129561] scsi host0: ata_piix [ 5.169893] scsi host1: ata_piix [ 5.171632] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.173920] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.930885] dracut-initqueue[579]: RTNETLINK answers: File exists [ 9.825548] random: crng init done [ 9.827226] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.266960] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.334071] printk: systemd: 23 output lines suppressed due to ratelimiting [ 11.562484] SELinux: Disabled at runtime. [ 11.619925] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.629595] systemd[1]: Detected virtualization kvm. [ 11.631760] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.099396] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.102803] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.107578] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.113368] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.117259] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.126237] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.130805] systemd[1]: proc-sys-fs-binfmt_misc.automount: Refusing to start, unit to trigger not loaded. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Remount Root and Kernel File Systems... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-getty.slice. Mounting Huge Pages File System... Mounting POSIX Message Queue File System... [ OK ] Reached target Paths. Starting udev Coldplug all Devices... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Mounting Kernel Debug File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Journal Service. [FAILED] Failed to [ 12.328407] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Device[ 12.565667] squashfs: version 4.0 (2009/01/31) Phillip Lougher s. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Mounted /mnt. [ OK ] Started udev Kernel Device Manager. [ 12.869290] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.898865] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.993449] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.013193] EDAC sbridge: Ver: 1.1.2 [ 14.249774] Key type dns_resolver registered [ 14.542049] NFS: Registering the id_resolver key type [ 14.546295] Key type id_resolver registered [ 14.547815] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... Starting Network Manager... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg357-client login: [ 69.538711] libcfs: loading out-of-tree module taints kernel. [ 69.657150] Key type ._llcrypt registered [ 69.661468] Key type .llcrypt registered [ 70.653747] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 70.672868] alg: No test for adler32 (adler32-zlib) [ 72.191421] Lustre: Lustre: Build Version: 2.17.57_44_g85e5bce [ 72.733192] hrtimer: interrupt took 6367826 ns [ 73.308166] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 75.175209] Key type lgssc registered [ 78.280980] Lustre: Echo OBD driver; http://www.lustre.org/ [ 203.021736] Lustre: Mounted lustre-client - version 2.17.57_44_g85e5bce [ 207.817935] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 224.619216] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing check_logdir /tmp/testlogs/ [ 228.831310] Lustre: lustre-OST0000-osc-ffff8a5dd1f43800: disconnect after 24s idle [ 229.409662] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing yml_node [ 233.604754] Lustre: DEBUG MARKER: Client: 2.17.57.44 [ 236.052382] Lustre: DEBUG MARKER: MDS: 2.17.57.44 [ 238.385352] Lustre: DEBUG MARKER: OSS: 2.17.57.44 [ 240.026658] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Fri Aug 21 15:03:20 EDT 2026 [ 255.627528] Lustre: Unmounted lustre-client [ 291.910275] Key type lgssc unregistered [ 292.171622] LNet: 8450:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 292.178754] LNetError: 8450:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 292.199045] LNet: Removed LNI 192.168.203.57@tcp [ 292.854120] Key type .llcrypt unregistered [ 292.856597] Key type ._llcrypt unregistered [ 315.293963] Key type ._llcrypt registered [ 315.296272] Key type .llcrypt registered [ 315.532980] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 315.543489] alg: No test for adler32 (adler32-zlib) [ 316.540703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 316.603130] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 316.624123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 321.166576] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 321.270972] LNet: 9330:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 321.277824] LNetError: 9330:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 321.324139] LNet: Removed LNI 192.168.203.57@tcp [ 322.442369] Key type .llcrypt unregistered [ 322.445171] Key type ._llcrypt unregistered [ 328.366731] Key type ._llcrypt registered [ 328.368597] Key type .llcrypt registered [ 329.802962] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 15:04:50 (1787339090) [ 330.071098] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 330.082576] alg: No test for adler32 (adler32-zlib) [ 330.950570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 331.044159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 331.067318] LNet: 9754:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 331.083523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 331.130402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 331.159465] LNet: 9761:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 337.171189] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 15:04:57 (1787339097) [ 337.361002] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 337.392205] LNet: 10339:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 337.414321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 337.488950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 343.193328] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 15:05:04 (1787339104) [ 343.317755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 343.341480] LNet: 10922:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 343.366096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 343.443795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 349.203582] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 15:05:09 (1787339109) [ 349.284468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 349.318247] LNet: 11504:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 349.345940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 349.455545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 354.999179] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 15:05:16 (1787339116) [ 355.190583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 355.205262] LNet: 12094:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 355.219267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 355.274502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 355.655612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 361.269555] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 15:05:22 (1787339122) [ 361.427767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 361.463238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 361.553945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 362.007801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 362.414958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 367.982339] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 15:05:29 (1787339129) [ 368.063483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 368.092335] LNet: 13276:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 368.099884] LNet: 13276:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 368.123430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 368.204211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 373.748839] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 15:05:34 (1787339134) [ 374.034968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 374.080173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 374.145168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 374.194032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 374.592971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 374.634468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 375.071543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 375.134829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 375.596135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 375.640406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 375.935863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 375.965358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 376.297901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 376.332814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 376.669767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 376.705692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 377.060814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 377.090245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 377.524821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 377.566699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 377.980815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 378.021300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 378.483449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 378.537162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 379.068502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 379.112406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 379.526844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 379.579848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 385.550851] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 15:05:46 (1787339146) [ 385.639478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 385.664456] LNet: 14541:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 385.676414] LNet: 14541:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 385.710366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 385.805599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 385.849443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 391.865415] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 15:05:52 (1787339152) [ 392.147745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 392.207621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 392.297660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 392.350654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 398.692528] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 15:05:59 (1787339159) [ 399.072504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 399.143720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 399.263252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 399.329910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 405.508199] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 15:06:06 (1787339166) [ 405.646242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 405.737655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 405.867390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 405.932027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 412.926263] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 15:06:13 (1787339173) [ 413.147289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 413.256723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 413.322433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 413.363575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 419.882971] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 15:06:20 (1787339180) [ 419.985769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 420.016195] LNet: 17472:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 420.024712] LNet: 17472:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 420.042567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 420.145762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 420.188508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 426.394235] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 15:06:27 (1787339187) [ 426.823896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 426.869637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 426.938412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 427.351909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 433.510169] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 15:06:34 (1787339194) [ 433.611516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 433.687203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 433.944842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 434.216620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 440.401825] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 15:06:41 (1787339201) [ 440.464806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 440.499975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 440.611806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 440.787360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 447.158682] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 15:06:47 (1787339207) [ 447.313550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 447.360538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 447.863381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 448.144510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 453.561359] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 15:06:54 (1787339214) [ 453.751367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 453.782684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 453.846324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 453.898818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 459.476607] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 15:07:00 (1787339220) [ 459.559178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 459.607636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 459.702911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 459.771202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 465.560648] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 15:07:06 (1787339226) [ 465.773362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 465.800915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 465.883893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 472.023819] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 15:07:12 (1787339232) [ 472.218588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 472.275671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 472.386447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 478.483258] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 15:07:19 (1787339239) [ 478.548634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 478.582879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 478.644660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 478.951789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 483.847414] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 15:07:25 (1787339245) [ 483.963604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 484.008940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 484.082413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 484.122568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 489.560582] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 15:07:30 (1787339250) [ 489.877530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 489.898919] LNet: 24090:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 489.904589] LNet: 24090:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 10 previous similar messages [ 489.925223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 490.000775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 490.034595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 495.829319] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 15:07:36 (1787339256) [ 495.921387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 495.974934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 496.073844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 496.113215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 501.456717] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 15:07:42 (1787339262) [ 501.549099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 501.607305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 501.738601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 501.778679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 508.103387] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 15:07:48 (1787339268) [ 508.215170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 508.246826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 508.292366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 508.366317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 508.380917] LNetError: 25862:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 508.427973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 508.467817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 513.913606] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 15:07:55 (1787339275) [ 514.195088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 514.223807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 514.291778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.mDYQghj5lt [ 514.344596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.mDYQghj5lt [ 514.393994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.mDYQghj5lt [ 519.450737] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 15:08:00 (1787339280) [ 519.609626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 519.681215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 519.789463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 519.821964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 519.885404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 526.461394] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 15:08:07 (1787339287) [ 526.624434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 526.717332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 526.868350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 526.913775] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 529.054052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.57@tcp [ 531.156948] Lustre: Lustre: Build Version: 2.17.57_44_g85e5bce [ 533.311467] Key type lgssc registered [ 534.777607] Lustre: Echo OBD driver; http://www.lustre.org/ [ 539.479848] LNetError: 27622:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.203.57@tcp99 has no available nets [ 543.213783] Key type lgssc unregistered [ 543.595762] LNetError: 28734:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 543.612802] LNet: Removed LNI 192.168.203.57@tcp [ 544.232756] Key type .llcrypt unregistered [ 544.235960] Key type ._llcrypt unregistered [ 550.540675] Key type ._llcrypt registered [ 550.542280] Key type .llcrypt registered [ 551.856597] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 15:08:32 (1787339312) [ 552.684936] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 552.736895] alg: No test for adler32 (adler32-zlib) [ 553.768479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 553.875250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 556.863242] LNet: Message forwarding will be enabled in 2s [ 559.969642] LNet: Message forwarding enabled [ 560.318478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 560.362504] LNet: Message forwarding will stop in 2s to 6s [ 565.006860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 566.284303] LNet: Message forwarding disabled [ 567.716582] LNet: Message forwarding will be enabled in 2s [ 571.423678] LNet: Message forwarding enabled [ 575.069925] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 15:08:56 (1787339336) [ 575.273466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 575.308803] LNet: 30142:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 576.521795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 576.590576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 581.643979] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 15:09:02 (1787339342) [ 581.809223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 581.844310] LNet: 30722:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 581.876935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 581.927225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 581.959086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 581.989153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 582.020525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 582.057797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 582.089694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 582.112798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 582.152515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 582.193365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 582.232769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 582.260368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 582.286389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 582.313794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 582.351994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 582.373881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 582.405618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 582.448590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 582.486904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 582.520828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 582.545676] LNetError: 30766:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 587.479916] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 15:09:08 (1787339348) [ 587.769647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 587.787663] LNet: 31337:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 587.811902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 587.885991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 593.161694] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 15:09:14 (1787339354) [ 594.792645] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg357-server [ 596.540934] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 15:09:17 (1787339357) [ 596.697981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 596.717865] LNet: 32267:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 596.736983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 596.787992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 596.811625] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 597.537800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.203.[58-62]@tcp [ 597.556466] LNet: 32400:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 597.604895] LNetError: 32271:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.60@tcp is being used as a gateway but routing feature is not turned on [ 597.627108] LNetError: 32271:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.60@tcp has gone from up to down [ 598.637312] LNetError: 32271:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.58@tcp is being used as a gateway but routing feature is not turned on [ 598.647643] LNetError: 32271:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 5 previous similar messages [ 599.664729] LNetError: 32271:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.58@tcp is being used as a gateway but routing feature is not turned on [ 599.679131] LNetError: 32271:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 2 previous similar messages [ 600.671867] LNetError: 32401:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.61@tcp has gone from up to down [ 600.718685] LNetError: 32401:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) Skipped 2 previous similar messages [ 601.772166] LNetError: 32271:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.58@tcp is being used as a gateway but routing feature is not turned on [ 601.785356] LNetError: 32271:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 5 previous similar messages [ 604.417126] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 15:09:25 (1787339365) [ 605.867519] LNetError: 32271:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.60@tcp is being used as a gateway but routing feature is not turned on [ 605.878822] LNetError: 32271:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 11 previous similar messages [ 606.184606] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg357-server [ 607.656460] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 15:09:28 (1787339368) [ 607.968072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 608.011937] LNet: 33337:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 608.019135] LNetError: 33337:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 608.045138] LNet: Removed LNI 192.168.203.57@tcp [ 608.079186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 608.146574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 608.175093] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 608.959739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.203.[58-63/2]@tcp [ 608.990264] LNet: 33454:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 609.014165] LNet: 33454:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 609.046446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.203.[58-63/2]@tcp [ 609.056023] LNetError: 33341:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp103 through 192.168.203.58@tcp has gone from up to down [ 609.092686] LNetError: 33341:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) Skipped 1 previous similar message [ 615.141632] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 15:09:36 (1787339376) [ 615.417463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 615.435190] LNetError: 34046:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 615.459711] LNet: Removed LNI 192.168.203.57@tcp [ 615.483302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 615.566174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 615.627604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 615.653861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 615.703622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 615.752863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 615.787753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 615.835826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 615.899198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 615.938909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 616.019776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 616.081473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 616.104159] LNet: 34099:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 616.110066] LNet: 34099:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 616.137474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 616.186767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 616.263736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 616.311744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 616.391560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 616.466620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 616.506207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 616.566618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 616.661506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 616.718454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 616.793434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 616.860272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 616.888387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 616.933564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 616.997186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 617.034272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 617.088419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 617.161737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 617.192836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 617.242740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 617.304974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 617.341520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 617.395243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 623.364112] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 15:09:44 (1787339384) [ 625.158524] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg357-server [ 626.650595] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 15:09:47 (1787339387) [ 628.617619] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg357-server [ 630.013975] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 15:09:51 (1787339391) [ 630.704982] Key type .llcrypt unregistered [ 630.709881] Key type ._llcrypt unregistered [ 636.376589] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 636.532284] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 636.538957] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 636.541434] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 638.969293] Key type ._llcrypt registered [ 638.972706] Key type .llcrypt registered [ 639.127493] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 639.137324] alg: No test for adler32 (adler32-zlib) [ 640.054396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 640.103322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 640.129294] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 640.599924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 640.617799] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 642.431906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 642.466963] LNet: Removed LNI 10.1.2.3@tcp [ 642.573767] LNet: 36172:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 642.580896] LNetError: 36172:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 642.601273] LNet: Removed LNI 192.168.203.57@tcp [ 643.193937] Key type .llcrypt unregistered [ 643.197229] Key type ._llcrypt unregistered [ 653.130194] Key type ._llcrypt registered [ 653.132377] Key type .llcrypt registered [ 654.165962] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 15:10:15 (1787339415) [ 654.587582] Key type .llcrypt unregistered [ 654.592341] Key type ._llcrypt unregistered [ 660.123248] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 660.296381] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 660.309941] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 660.312249] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 662.920922] Key type ._llcrypt registered [ 662.924423] Key type .llcrypt registered [ 663.176798] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 663.197040] alg: No test for adler32 (adler32-zlib) [ 664.162972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 664.246339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 664.287199] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 665.099265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 665.144752] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 667.282923] LNet: 37672:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 667.291316] LNetError: 37672:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 667.295299] LNet: Removed LNI 10.1.2.3@tcp [ 667.319194] LNet: Removed LNI 192.168.203.57@tcp [ 667.790440] Key type .llcrypt unregistered [ 667.793930] Key type ._llcrypt unregistered [ 677.289482] Key type ._llcrypt registered [ 677.294918] Key type .llcrypt registered [ 678.796452] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 15:10:39 (1787339439) [ 679.345321] Key type .llcrypt unregistered [ 679.350629] Key type ._llcrypt unregistered [ 684.804650] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 684.984925] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 684.994932] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 685.018263] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 687.640121] Key type ._llcrypt registered [ 687.643692] Key type .llcrypt registered [ 687.867682] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 687.895298] alg: No test for adler32 (adler32-zlib) [ 688.860444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 688.954636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 688.984582] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 691.339883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 691.373274] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 693.641732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 693.673131] LNet: Removed LNI 10.1.2.3@tcp [ 693.697860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 693.746387] LNet: Removed LNI 10.1.2.31@tcp [ 693.770884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 693.811705] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 694.861477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 694.891962] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 696.033195] LNet: 39601:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 696.037275] LNetError: 39601:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 696.046515] LNet: Removed LNI 10.1.2.31@tcp [ 696.062177] LNet: Removed LNI 10.1.2.3@tcp [ 696.765903] Key type .llcrypt unregistered [ 696.779803] Key type ._llcrypt unregistered [ 708.770520] Key type ._llcrypt registered [ 708.772058] Key type .llcrypt registered [ 710.072889] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 15:11:10 (1787339470) [ 710.508871] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 710.528142] alg: No test for adler32 (adler32-zlib) [ 711.394414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 711.437554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 711.459456] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 712.336636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 712.350989] LNet: 40557:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 712.357601] LNetError: 40557:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 712.375407] LNet: Removed LNI 192.168.203.57@tcp [ 712.396286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 712.459660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.203.57@tcp [ 712.521442] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 713.181909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.203.57@tcp [ 713.222463] LNet: Removed LNI 192.168.203.57@tcp [ 713.278877] LNet: 40692:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 713.951419] Key type .llcrypt unregistered [ 713.954302] Key type ._llcrypt unregistered [ 727.514096] Key type ._llcrypt registered [ 727.517594] Key type .llcrypt registered [ 729.199671] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 15:11:29 (1787339489) [ 731.406566] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg357-server [ 733.301899] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 15:11:34 (1787339494) [ 733.959445] Key type .llcrypt unregistered [ 733.963109] Key type ._llcrypt unregistered [ 740.123772] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 740.314826] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 740.319478] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 740.326635] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 742.834064] Key type ._llcrypt registered [ 742.838253] Key type .llcrypt registered [ 743.023630] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 743.039851] alg: No test for adler32 (adler32-zlib) [ 743.918120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 743.980651] LNet: Added LNI 192.168.203.57@tcp1 [8/256/0/180] [ 744.564620] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 746.480229] LNet: 42536:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 746.488617] LNetError: 42536:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 746.494660] LNet: Removed LNI 192.168.203.57@tcp1 [ 746.501796] LNet: Removed LNI 10.1.2.3@tcp2 [ 747.257611] Key type .llcrypt unregistered [ 747.260638] Key type ._llcrypt unregistered [ 758.903265] Key type ._llcrypt registered [ 758.907453] Key type .llcrypt registered [ 759.932933] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 15:12:00 (1787339520) [ 760.661462] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 760.678090] alg: No test for adler32 (adler32-zlib) [ 761.710439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 761.812269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 761.863675] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 762.913376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 762.936969] LNet: 43513:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 762.946898] LNetError: 43513:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 762.955788] LNet: Removed LNI 192.168.203.57@tcp [ 762.970532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 763.026891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 763.059908] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 769.772516] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 15:12:10 (1787339530) [ 770.092461] LNet: 44104:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 770.103898] LNetError: 44104:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 770.118637] LNet: Removed LNI 192.168.203.57@tcp [ 770.819460] Key type .llcrypt unregistered [ 770.823149] Key type ._llcrypt unregistered [ 778.168782] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 778.388591] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 778.401532] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 778.416959] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 779.803293] Key type ._llcrypt registered [ 779.806461] Key type .llcrypt registered [ 779.991759] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 780.002709] alg: No test for adler32 (adler32-zlib) [ 780.936359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 781.027856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 781.058262] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 781.680322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 781.696388] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 783.523357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 783.557324] LNet: 44840:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 783.565980] LNetError: 44840:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 783.582093] LNet: Removed LNI 10.1.2.3@tcp [ 783.612139] LNet: Removed LNI 192.168.203.57@tcp [ 783.632788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 783.746323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 783.787083] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 783.798439] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 784.476875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 784.511554] LNet: 44867:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 784.516882] LNetError: 44867:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 784.522967] LNetError: 44867:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 784.529533] LNet: Removed LNI 10.1.2.3@tcp [ 784.553428] LNet: Removed LNI 192.168.203.57@tcp [ 784.592838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 784.672370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 784.712985] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 784.718836] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 785.330308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 785.372828] LNet: Added LNI 192.168.203.57@tcp2 [8/256/0/180] [ 786.433876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 786.459616] LNet: 45029:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 786.465942] LNetError: 45029:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 786.474292] LNetError: 45029:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 786.482299] LNet: Removed LNI 10.1.2.3@tcp [ 786.486225] LNet: Removed LNI 192.168.203.57@tcp [ 786.498893] LNet: Removed LNI 192.168.203.57@tcp2 [ 786.518637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 786.595644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 786.632269] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 786.648996] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 786.654153] LNet: Added LNI 192.168.203.57@tcp2 [8/256/0/180] [ 792.973735] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 15:12:33 (1787339553) [ 793.262871] LNet: 45626:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 793.278308] LNetError: 45626:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 793.285239] LNetError: 45626:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 793.296147] LNet: Removed LNI 10.1.2.3@tcp [ 793.300592] LNet: Removed LNI 192.168.203.57@tcp [ 793.310989] LNet: Removed LNI 192.168.203.57@tcp2 [ 793.877728] Key type .llcrypt unregistered [ 793.885230] Key type ._llcrypt unregistered [ 799.356660] Key type ._llcrypt registered [ 799.362422] Key type .llcrypt registered [ 799.543720] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 799.554336] alg: No test for adler32 (adler32-zlib) [ 800.476154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 800.574144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 800.605787] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 801.308463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 801.323329] LNet: 46025:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 801.330143] LNetError: 46025:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 801.351163] LNet: Removed LNI 192.168.203.57@tcp [ 801.378117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 801.442945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 801.487364] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 802.009657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 802.034780] LNet: 46055:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 802.043881] LNetError: 46055:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 802.069181] LNet: Removed LNI 192.168.203.57@tcp [ 802.111960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 802.178813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 802.213677] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 802.691333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 802.740395] LNet: Removed LNI 192.168.203.57@tcp [ 802.753597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 802.816285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 802.845254] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 803.324225] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 803.513626] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 803.521051] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 803.530566] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 805.932871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 805.973880] LNet: 46161:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 805.981323] LNet: 46161:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 805.988362] LNetError: 46161:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 805.995271] LNetError: 46161:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 806.006545] LNet: Removed LNI 192.168.203.57@tcp [ 806.057262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 806.175661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 806.224507] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 812.446838] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 15:12:53 (1787339573) [ 812.690511] LNet: 46758:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 812.701579] LNetError: 46758:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 812.720569] LNet: Removed LNI 192.168.203.57@tcp [ 813.378650] Key type .llcrypt unregistered [ 813.381241] Key type ._llcrypt unregistered [ 818.951348] Key type ._llcrypt registered [ 818.953792] Key type .llcrypt registered [ 819.252144] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 819.287518] alg: No test for adler32 (adler32-zlib) [ 820.238811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 820.280449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 820.306458] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 820.990624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 821.012622] LNet: 47161:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 821.018641] LNetError: 47161:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 821.028787] LNet: Removed LNI 192.168.203.57@tcp [ 821.050052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 821.124964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 821.142907] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 827.589092] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 15:13:08 (1787339588) [ 827.681975] LNet: 47744:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 827.688607] LNetError: 47744:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 827.712896] LNet: Removed LNI 192.168.203.57@tcp [ 828.328078] Key type .llcrypt unregistered [ 828.334645] Key type ._llcrypt unregistered [ 834.918783] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 835.055893] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 835.062026] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 835.072034] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 837.657169] Key type ._llcrypt registered [ 837.659035] Key type .llcrypt registered [ 837.873608] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 837.892035] alg: No test for adler32 (adler32-zlib) [ 838.795867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 838.879464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 838.911947] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 839.828823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 839.854554] LNet: 48207:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 839.860887] LNetError: 48207:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 839.879893] LNet: Removed LNI 192.168.203.57@tcp [ 839.913638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 839.970966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 840.018796] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 846.378949] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 15:13:26 (1787339606) [ 846.543528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 846.590786] LNet: 48801:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 846.601275] LNetError: 48801:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 846.621027] LNet: Removed LNI 192.168.203.57@tcp [ 846.644878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 846.829202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 853.320312] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 15:13:33 (1787339613) [ 853.526755] LNet: 49386:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 854.083678] Key type .llcrypt unregistered [ 854.086934] Key type ._llcrypt unregistered [ 860.171343] Key type ._llcrypt registered [ 860.173718] Key type .llcrypt registered [ 860.418161] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 860.428532] alg: No test for adler32 (adler32-zlib) [ 861.369454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 861.418034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 861.435829] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 862.210684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 862.235535] LNet: 49789:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 862.250106] LNetError: 49789:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 862.268413] LNet: Removed LNI 192.168.203.57@tcp [ 862.297386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 862.395435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 862.451427] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 868.860572] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 15:13:49 (1787339629) [ 868.998461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 869.020302] LNet: 50382:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 869.031888] LNetError: 50382:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 869.046229] LNet: Removed LNI 192.168.203.57@tcp [ 869.070510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 869.144579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 874.527295] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 15:13:55 (1787339635) [ 876.192974] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 877.794931] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 15:13:58 (1787339638) [ 879.280720] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 880.818752] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 15:14:01 (1787339641) [ 882.503524] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 884.059605] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 15:14:05 (1787339645) [ 885.678711] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 887.602099] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 15:14:08 (1787339648) [ 889.157720] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 890.743435] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 15:14:11 (1787339651) [ 891.012876] LNet: 52434:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 891.931610] Key type .llcrypt unregistered [ 891.935412] Key type ._llcrypt unregistered [ 898.924923] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 899.065655] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 899.069922] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 899.090948] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 901.740062] Key type ._llcrypt registered [ 901.746637] Key type .llcrypt registered [ 901.929548] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 901.945656] alg: No test for adler32 (adler32-zlib) [ 902.903400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 903.003747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 903.032608] LNet: Added LNI 192.168.203.57@tcp [8/512/0/180] [ 903.038385] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 903.041681] LNet: 52760:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 903.151904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 903.175373] LNet: 52779:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 903.191622] LNetError: 52779:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 903.206672] LNet: Removed LNI 10.1.2.3@tcp [ 903.210683] LNetError: 52779:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 903.240123] LNet: Removed LNI 192.168.203.57@tcp [ 903.266307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 903.335985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 903.358963] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 903.431307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 903.452437] LNet: 52803:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 903.456896] LNetError: 52803:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 903.471734] LNet: Removed LNI 192.168.203.57@tcp [ 903.493563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 903.553933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 903.584199] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 903.589434] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 903.595563] LNet: 52811:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 903.681862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 903.707781] LNet: 52830:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 903.716388] LNetError: 52830:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 903.726834] LNet: Removed LNI 10.1.2.3@tcp [ 903.730830] LNetError: 52830:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 903.745527] LNet: Removed LNI 192.168.203.57@tcp [ 903.776250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 903.816500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 903.831304] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 903.834634] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 903.836335] LNet: 52838:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 903.969481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 903.984185] LNet: 52864:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 903.993527] LNetError: 52864:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 903.998629] LNet: Removed LNI 10.1.2.3@tcp [ 904.000865] LNetError: 52864:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 904.021780] LNet: Removed LNI 192.168.203.57@tcp [ 904.043875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 904.139279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 904.170331] LNet: Added LNI 192.168.203.57@tcp [8/256/0/182] [ 904.179431] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 904.282327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 904.299440] LNet: 52892:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 904.303842] LNetError: 52892:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 904.311901] LNet: Removed LNI 10.1.2.3@tcp [ 904.316742] LNetError: 52892:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 904.337794] LNet: Removed LNI 192.168.203.57@tcp [ 904.375154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 904.464234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 904.497960] LNet: Added LNI 192.168.203.57@tcp [8/256/0/182] [ 904.510150] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 904.516734] LNet: 52900:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 904.581130] LNet: Removed LNI 10.1.2.3@tcp [ 904.591114] LNet: Removed LNI 192.168.203.57@tcp [ 905.323534] Key type .llcrypt unregistered [ 905.333425] Key type ._llcrypt unregistered [ 918.020381] Key type ._llcrypt registered [ 918.022059] Key type .llcrypt registered [ 919.230294] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 15:14:39 (1787339679) [ 920.450145] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 921.833965] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 15:14:42 (1787339682) [ 922.156901] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 922.172524] alg: No test for adler32 (adler32-zlib) [ 923.036877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 923.076417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 923.089570] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 923.627534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.203.57 [ 923.647536] LNet: 54170:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 923.656032] LNet: Removed LNI 192.168.203.57@tcp [ 929.691707] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 15:14:50 (1787339690) [ 929.755514] LNet: 54734:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 929.760894] LNetError: 54734:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 929.773471] LNet: Removed LNI 192.168.203.57@tcp [ 930.565364] Key type .llcrypt unregistered [ 930.568439] Key type ._llcrypt unregistered [ 935.971521] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 936.117462] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 938.629823] Key type ._llcrypt registered [ 938.631946] Key type .llcrypt registered [ 938.791158] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 938.819769] alg: No test for adler32 (adler32-zlib) [ 940.061834] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 945.389882] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 15:15:06 (1787339706) [ 945.600721] LNet: 55652:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 945.612731] LNetError: 55652:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 945.638373] LNet: Removed LNI 10.1.2.3@tcp [ 946.276122] Key type .llcrypt unregistered [ 946.277896] Key type ._llcrypt unregistered [ 951.575759] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 951.781125] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 951.787171] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 951.790853] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 954.389579] Key type ._llcrypt registered [ 954.392562] Key type .llcrypt registered [ 954.504297] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 954.516496] alg: No test for adler32 (adler32-zlib) [ 955.443126] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 961.175865] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 15:15:22 (1787339722) [ 962.706357] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 964.520504] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 15:15:25 (1787339725) [ 964.805445] LNet: 56844:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 964.813739] LNetError: 56844:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 964.827612] LNet: Removed LNI 10.1.2.3@tcp [ 965.479519] Key type .llcrypt unregistered [ 965.483384] Key type ._llcrypt unregistered [ 970.854870] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 971.044732] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 971.049540] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 971.058326] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 973.744645] Key type ._llcrypt registered [ 973.747623] Key type .llcrypt registered [ 973.963430] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 973.983080] alg: No test for adler32 (adler32-zlib) [ 974.928739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 975.019316] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 982.436956] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 15:15:43 (1787339743) [ 983.043240] LNet: 58077:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 983.053304] LNetError: 58077:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 983.068484] LNet: Removed LNI 10.1.2.3@tcp [ 983.773979] Key type .llcrypt unregistered [ 983.776157] Key type ._llcrypt unregistered [ 991.140943] Key type ._llcrypt registered [ 991.142958] Key type .llcrypt registered [ 991.409890] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 991.431475] alg: No test for adler32 (adler32-zlib) [ 992.579737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 992.647797] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1000.848523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.157@tcp [ 1001.563843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp -m GET -e local_interrupt -r 1 [ 1001.621316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp -m GET -e local_interrupt -r 1 [ 1001.668608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.157@tcp [ 1001.702642] LNet: There was an unexpected network error while writing to 192.168.203.157: rc = -22 [ 1001.892336] LNet: 1 local NIs in recovery (showing 1): 192.168.203.57@tcp [ 1001.913955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1001.947394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1001.972848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1002.002718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1002.036774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1002.066856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1002.102979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1002.147677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1002.174984] LNet: 58534:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1002.207711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1002.255948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1002.310873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1002.361279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1002.398021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1002.430896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1002.475757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1002.997694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp -m GET -e local_dropped -r 1 [ 1003.062635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp -m GET -e local_dropped -r 1 [ 1003.133987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.157@tcp [ 1003.165749] LNet: There was an unexpected network error while writing to 192.168.203.157: rc = -22 [ 1003.450185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1003.479884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1003.511752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1003.546494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1003.579374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1003.618779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1003.676334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1003.734263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1003.764112] LNet: 58643:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1003.780458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1003.813980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1003.846371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1003.872272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1003.904113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1003.946818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1003.987759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1004.604990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp -m GET -e local_aborted -r 1 [ 1004.655554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp -m GET -e local_aborted -r 1 [ 1004.704564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.157@tcp [ 1004.725519] LNet: There was an unexpected network error while writing to 192.168.203.157: rc = -22 [ 1004.942718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1004.982331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1005.033206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1005.076656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1005.117666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1005.146224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1005.185815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1005.216390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1005.230390] LNet: 58752:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1005.246618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1005.294518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1005.338096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1005.416195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1005.458574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1005.496360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1005.535983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1006.122402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp -m GET -e local_no_route -r 1 [ 1006.206070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp -m GET -e local_no_route -r 1 [ 1006.272398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.157@tcp [ 1006.584057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1006.623935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1006.696225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1006.739896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1006.768848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1006.819751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1006.847799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1006.886672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1006.909329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1006.936217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1006.972943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1007.016384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1007.057675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1007.088142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1007.122053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1007.784779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp -m GET -e local_timeout -r 1 [ 1007.823744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp -m GET -e local_timeout -r 1 [ 1007.853898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.157@tcp [ 1007.874937] LNet: There was an unexpected network error while writing to 192.168.203.157: rc = -22 [ 1007.879314] LNet: Skipped 2 previous similar messages [ 1008.051453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1008.088598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1008.126562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1008.149767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1008.183515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1008.222634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1008.284687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1008.343302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1008.368562] LNet: 58971:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1008.376717] LNet: 58971:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1008.397897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1008.451469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1008.493700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1008.527361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1008.579749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1008.633275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1008.685151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1009.171342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp -m GET -e local_error -r 1 [ 1009.211147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp -m GET -e local_error -r 1 [ 1009.251788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.157@tcp [ 1009.531511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1009.566778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1009.608960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1009.644385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1009.675650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1009.707773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1009.740868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1009.771852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1009.812439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1009.840930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1009.881417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1009.919632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1009.964445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1010.012425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1010.070611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1010.454971] LNet: 59125:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1010.462602] LNetError: 59125:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1010.481895] LNet: Removed LNI 192.168.203.57@tcp [ 1010.944891] Key type .llcrypt unregistered [ 1010.946815] Key type ._llcrypt unregistered [ 1030.757956] Key type ._llcrypt registered [ 1030.762864] Key type .llcrypt registered [ 1031.855276] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 15:16:32 (1787339792) [ 1032.985728] Key type .llcrypt unregistered [ 1032.987481] Key type ._llcrypt unregistered [ 1039.809575] Key type ._llcrypt registered [ 1039.813105] Key type .llcrypt registered [ 1040.105099] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1040.127268] alg: No test for adler32 (adler32-zlib) [ 1041.019533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1041.058798] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1048.668172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.157@tcp [ 1058.474676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1058.531430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1058.575890] LNet: Added LNI 192.168.203.57@tcp1 [8/256/0/180] [ 1059.909558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp -m GET -e local_interrupt -r 1 [ 1059.939789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp1 -m GET -e local_interrupt -r 1 [ 1059.977663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp -m GET -e local_interrupt -r 1 [ 1060.014371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp1 -m GET -e local_interrupt -r 1 [ 1060.076957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp -m GET -e local_interrupt -r 1 [ 1060.134310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp1 -m GET -e local_interrupt -r 1 [ 1060.200362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp -m GET -e local_interrupt -r 1 [ 1060.240986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp1 -m GET -e local_interrupt -r 1 [ 1060.288590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.157@tcp [ 1060.315335] LNet: There was an unexpected network error while writing to 192.168.203.157: rc = -22 [ 1060.321581] LNet: 1 local NIs in recovery (showing 1): 192.168.203.57@tcp [ 1060.549731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1060.585673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1060.620753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1060.669279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1060.711369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1060.739665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1060.770860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1060.819703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1060.831702] LNet: 60596:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1060.848617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1060.873303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1060.894716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1060.920433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1060.951984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1060.982880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1061.023194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1061.542396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp -m GET -e local_dropped -r 1 [ 1061.585056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp1 -m GET -e local_dropped -r 1 [ 1061.617879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp -m GET -e local_dropped -r 1 [ 1061.657567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp1 -m GET -e local_dropped -r 1 [ 1061.689667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp -m GET -e local_dropped -r 1 [ 1061.735228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp1 -m GET -e local_dropped -r 1 [ 1061.769485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp -m GET -e local_dropped -r 1 [ 1061.799577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp1 -m GET -e local_dropped -r 1 [ 1061.840815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.157@tcp [ 1061.858211] LNet: There was an unexpected network error while writing to 192.168.203.157: rc = -22 [ 1061.862732] LNet: Skipped 2 previous similar messages [ 1062.076592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1062.112276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1062.142080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1062.167381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1062.196611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1062.249940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1062.285840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1062.331430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1062.357596] LNet: 60717:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1062.384564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1062.429445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1062.460824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1062.497878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1062.536374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1062.562095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1062.597880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1063.056419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp -m GET -e local_aborted -r 1 [ 1063.082964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp1 -m GET -e local_aborted -r 1 [ 1063.113619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp -m GET -e local_aborted -r 1 [ 1063.150567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp1 -m GET -e local_aborted -r 1 [ 1063.183560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp -m GET -e local_aborted -r 1 [ 1063.214929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp1 -m GET -e local_aborted -r 1 [ 1063.258682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp -m GET -e local_aborted -r 1 [ 1063.286013] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp1 -m GET -e local_aborted -r 1 [ 1063.310840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.157@tcp [ 1063.329428] LNet: There was an unexpected network error while writing to 192.168.203.157: rc = -22 [ 1063.337499] LNet: Skipped 2 previous similar messages [ 1063.558829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1063.596618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1063.629743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1063.664322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1063.706318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1063.743382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1063.774831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1063.800792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1063.824455] LNet: 60839:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1063.847512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1063.891825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1063.930813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1063.955833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1064.020416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1064.074790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1064.116471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1064.625109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp -m GET -e local_no_route -r 1 [ 1064.652565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp1 -m GET -e local_no_route -r 1 [ 1064.688036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp -m GET -e local_no_route -r 1 [ 1064.727237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp1 -m GET -e local_no_route -r 1 [ 1064.772809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp -m GET -e local_no_route -r 1 [ 1064.816703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp1 -m GET -e local_no_route -r 1 [ 1064.874650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp -m GET -e local_no_route -r 1 [ 1064.927361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp1 -m GET -e local_no_route -r 1 [ 1064.956711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.157@tcp [ 1065.211767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1065.241171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1065.276699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1065.304786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1065.344244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1065.399380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1065.443284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1065.472221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1065.527236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1065.571730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1065.619428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1065.664352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1065.727512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1065.769121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1065.802582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1066.391725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp -m GET -e local_timeout -r 1 [ 1066.434431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp1 -m GET -e local_timeout -r 1 [ 1066.474534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp -m GET -e local_timeout -r 1 [ 1066.503649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp1 -m GET -e local_timeout -r 1 [ 1066.539442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp -m GET -e local_timeout -r 1 [ 1066.572720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp1 -m GET -e local_timeout -r 1 [ 1066.605754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp -m GET -e local_timeout -r 1 [ 1066.639427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp1 -m GET -e local_timeout -r 1 [ 1066.669613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.157@tcp [ 1066.692415] LNet: There was an unexpected network error while writing to 192.168.203.157: rc = -22 [ 1066.696580] LNet: Skipped 5 previous similar messages [ 1066.905720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1066.936423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1066.996861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1067.046296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1067.082050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1067.133645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1067.183581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1067.241360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1067.283899] LNet: 61081:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1067.291866] LNet: 61081:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1067.318761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1067.352354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1067.382384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1067.416814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1067.445683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1067.486859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1067.512521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1067.894471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp -m GET -e local_error -r 1 [ 1067.922358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp1 -m GET -e local_error -r 1 [ 1067.948194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp -m GET -e local_error -r 1 [ 1067.979717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp1 -m GET -e local_error -r 1 [ 1068.022985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp -m GET -e local_error -r 1 [ 1068.060502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp1 -m GET -e local_error -r 1 [ 1068.102598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp -m GET -e local_error -r 1 [ 1068.142113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp1 -m GET -e local_error -r 1 [ 1068.181173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.157@tcp [ 1068.412628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1068.447074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1068.479431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1068.516295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1068.548774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1068.587286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1068.610995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1068.647724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1068.693099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1068.715823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1068.750782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1068.776065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1068.816438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1068.845280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1068.869740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1073.966931] LNet: 61282:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1073.975842] LNetError: 61282:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1073.989419] LNet: Removed LNI 192.168.203.57@tcp [ 1074.001944] LNet: Removed LNI 192.168.203.57@tcp1 [ 1074.356699] Key type .llcrypt unregistered [ 1074.361543] Key type ._llcrypt unregistered [ 1090.270298] Key type ._llcrypt registered [ 1090.274734] Key type .llcrypt registered [ 1091.508210] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 15:17:32 (1787339852) [ 1092.095512] Key type .llcrypt unregistered [ 1092.097890] Key type ._llcrypt unregistered [ 1097.253566] Key type ._llcrypt registered [ 1097.255560] Key type .llcrypt registered [ 1097.449751] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1097.469558] alg: No test for adler32 (adler32-zlib) [ 1098.346427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1098.373852] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1105.938814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.157@tcp [ 1106.474428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp -m GET -e remote_dropped -r 1 [ 1106.508700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp -m GET -e remote_dropped -r 1 [ 1106.540767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.157@tcp [ 1106.567994] LNet: There was an unexpected network error while writing to 192.168.203.157: rc = -22 [ 1106.800418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1106.828678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1106.852544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1106.902820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1106.946392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1106.979277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1107.027437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1107.060591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1107.086073] LNet: 62535:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1107.109504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1107.147905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1107.181806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1107.218806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1107.249618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1107.280938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1107.325830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1107.809620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp -m GET -e remote_error -r 1 [ 1107.837386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp -m GET -e remote_error -r 1 [ 1107.881866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.157@tcp [ 1107.895977] LNet: There was an unexpected network error while writing to 192.168.203.157: rc = -22 [ 1108.137861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1108.174185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1108.205077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1108.232484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1108.264623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1108.302489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1108.355471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1108.381830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1108.412835] LNet: 62644:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1108.444188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1108.501540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1108.528979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1108.550302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1108.584161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1108.642450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1108.684918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1109.193472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp -m GET -e remote_timeout -r 1 [ 1109.225427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp -m GET -e remote_timeout -r 1 [ 1109.252763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.157@tcp [ 1109.274445] LNet: There was an unexpected network error while writing to 192.168.203.157: rc = -22 [ 1109.474868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1109.523629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1109.563152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1109.603482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1109.637841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1109.672461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1109.712278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1109.758510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1109.801393] LNet: 62753:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1109.841202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1109.887585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1109.936414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1109.978334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1110.020846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1110.055243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1110.082914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1110.334421] LNet: 62798:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1110.339898] LNetError: 62798:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1110.360456] LNet: Removed LNI 192.168.203.57@tcp [ 1110.761488] Key type .llcrypt unregistered [ 1110.765192] Key type ._llcrypt unregistered [ 1128.279376] Key type ._llcrypt registered [ 1128.281835] Key type .llcrypt registered [ 1129.839784] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 15:18:10 (1787339890) [ 1130.946360] Key type .llcrypt unregistered [ 1130.953859] Key type ._llcrypt unregistered [ 1137.823965] Key type ._llcrypt registered [ 1137.827908] Key type .llcrypt registered [ 1138.149521] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1138.183885] alg: No test for adler32 (adler32-zlib) [ 1139.080521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1139.115991] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1146.259856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.157@tcp [ 1154.225843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1154.273853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1154.317253] LNet: Added LNI 192.168.203.57@tcp1 [8/256/0/180] [ 1155.842409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp -m GET -e remote_dropped -r 1 [ 1155.904723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp1 -m GET -e remote_dropped -r 1 [ 1155.948365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp -m GET -e remote_dropped -r 1 [ 1155.996785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp1 -m GET -e remote_dropped -r 1 [ 1156.044829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp -m GET -e remote_dropped -r 1 [ 1156.077773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp1 -m GET -e remote_dropped -r 1 [ 1156.109289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp -m GET -e remote_dropped -r 1 [ 1156.137892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp1 -m GET -e remote_dropped -r 1 [ 1156.170503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.157@tcp [ 1156.198200] LNet: There was an unexpected network error while writing to 192.168.203.157: rc = -22 [ 1156.205287] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.157@tcp [ 1156.424477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1156.465219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1156.512339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1156.545512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1156.598692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1156.645042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1156.697068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1156.732827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1156.751795] LNet: 64282:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1156.769908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1156.795697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1156.830854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1156.872777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1156.915833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1156.957715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1157.019512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1157.062263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1157.536951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp -m GET -e remote_error -r 1 [ 1157.573076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp1 -m GET -e remote_error -r 1 [ 1157.612244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp -m GET -e remote_error -r 1 [ 1157.650650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp1 -m GET -e remote_error -r 1 [ 1157.707991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp -m GET -e remote_error -r 1 [ 1157.758380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp1 -m GET -e remote_error -r 1 [ 1157.800461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp -m GET -e remote_error -r 1 [ 1157.831643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp1 -m GET -e remote_error -r 1 [ 1157.868708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.157@tcp [ 1157.891994] LNet: There was an unexpected network error while writing to 192.168.203.157: rc = -22 [ 1157.896815] LNet: Skipped 2 previous similar messages [ 1158.084109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1158.105738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1158.128793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1158.153717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1158.176964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1158.208415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1158.238667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1158.270627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1158.295977] LNet: 64405:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1158.324853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1158.357782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1158.396221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1158.433899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1158.469979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1158.505691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1158.528122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1158.566534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1159.202767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp -m GET -e remote_timeout -r 1 [ 1159.234663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp1 -m GET -e remote_timeout -r 1 [ 1159.263970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp -m GET -e remote_timeout -r 1 [ 1159.318192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp1 -m GET -e remote_timeout -r 1 [ 1159.387251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp -m GET -e remote_timeout -r 1 [ 1159.436785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp1 -m GET -e remote_timeout -r 1 [ 1159.474791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp -m GET -e remote_timeout -r 1 [ 1159.516253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp1 -m GET -e remote_timeout -r 1 [ 1159.548534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.157@tcp [ 1159.566956] LNet: There was an unexpected network error while writing to 192.168.203.157: rc = -22 [ 1159.576763] LNet: Skipped 1 previous similar message [ 1159.766501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1159.805751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1159.833766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1159.874617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1159.909715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1159.937910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1159.966323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1160.000375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1160.021379] LNet: 64529:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1160.040908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1160.073374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1160.123270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1160.152415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1160.182857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1160.229827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1160.273152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1160.329756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1165.441958] LNet: 64611:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1165.450655] LNetError: 64611:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1165.458913] LNet: Removed LNI 192.168.203.57@tcp [ 1165.483239] LNet: Removed LNI 192.168.203.57@tcp1 [ 1165.940844] Key type .llcrypt unregistered [ 1165.945962] Key type ._llcrypt unregistered [ 1181.145921] Key type ._llcrypt registered [ 1181.153559] Key type .llcrypt registered [ 1182.335905] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 15:19:03 (1787339943) [ 1183.924697] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1185.862756] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 15:19:06 (1787339946) [ 1186.939479] Key type .llcrypt unregistered [ 1186.942562] Key type ._llcrypt unregistered [ 1192.471118] Key type ._llcrypt registered [ 1192.476525] Key type .llcrypt registered [ 1192.714373] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1192.735820] alg: No test for adler32 (adler32-zlib) [ 1193.727847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1193.779550] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1201.467661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.157@tcp [ 1201.997592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp -m GET -e network_timeout -r 1 [ 1202.025322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp -m GET -e network_timeout -r 1 [ 1202.059978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.157@tcp [ 1202.075420] LNet: There was an unexpected network error while writing to 192.168.203.157: rc = -22 [ 1202.256859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1202.286603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1202.318564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1202.351674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1202.384060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1202.420979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1202.454975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1202.493222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1202.516825] LNet: 66160:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1202.533502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1202.576341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1202.603166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1202.635340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1202.668145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1202.699672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1202.922695] LNet: 66203:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1202.933352] LNetError: 66203:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1202.949639] LNet: Removed LNI 192.168.203.57@tcp [ 1203.264526] Key type .llcrypt unregistered [ 1203.270414] Key type ._llcrypt unregistered [ 1218.400937] Key type ._llcrypt registered [ 1218.404143] Key type .llcrypt registered [ 1218.600957] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1218.626259] alg: No test for adler32 (adler32-zlib) [ 1219.664501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1219.708470] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1227.012677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.157@tcp [ 1234.820870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1234.858711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1234.917145] LNet: Added LNI 192.168.203.57@tcp1 [8/256/0/180] [ 1236.000487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp -m GET -e network_timeout -r 1 [ 1236.031074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp1 -m GET -e network_timeout -r 1 [ 1236.076710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp -m GET -e network_timeout -r 1 [ 1236.111933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp1 -m GET -e network_timeout -r 1 [ 1236.144673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp -m GET -e network_timeout -r 1 [ 1236.170396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp1 -m GET -e network_timeout -r 1 [ 1236.198389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp -m GET -e network_timeout -r 1 [ 1236.229257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp1 -m GET -e network_timeout -r 1 [ 1236.268528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.157@tcp [ 1236.294122] LNet: There was an unexpected network error while writing to 192.168.203.157: rc = -22 [ 1236.493090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1236.524613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1236.553742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1236.596155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1236.624101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1236.659397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1236.696516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1236.721475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1236.734568] LNet: 67085:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1236.757368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1236.801629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1236.839187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1236.878256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1236.902725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1236.925856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1240.618607] LNet: There was an unexpected network error while writing to 192.168.203.157: rc = -22 [ 1241.034643] LNet: 67163:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1241.051749] LNetError: 67163:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1241.057864] LNet: Removed LNI 192.168.203.57@tcp [ 1241.077450] LNet: Removed LNI 192.168.203.57@tcp1 [ 1241.425562] Key type .llcrypt unregistered [ 1241.428416] Key type ._llcrypt unregistered [ 1256.230734] Key type ._llcrypt registered [ 1256.233430] Key type .llcrypt registered [ 1257.600175] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 15:20:18 (1787340018) [ 1258.023784] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1258.048673] alg: No test for adler32 (adler32-zlib) [ 1258.970414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1259.023661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1259.042917] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1259.761727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1259.795205] LNet: Added LNI 192.168.203.57@tcp1 [8/256/0/180] [ 1260.566737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.57@tcp [ 1260.632232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1260.674799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1260.705439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1260.764488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1260.804162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1260.840497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.57@tcp [ 1260.863361] LNetError: 68263:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.57@tcp to recovery [ 1261.023310] LNet: 1 local NIs in recovery (showing 1): 192.168.203.57@tcp [ 1261.024300] LNet: There was an unexpected network error while writing to 192.168.203.57: rc = -22 [ 1263.072185] LNet: There was an unexpected network error while writing to 192.168.203.57: rc = -22 [ 1263.266612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1267.169528] LNet: There was an unexpected network error while writing to 192.168.203.57: rc = -22 [ 1267.743915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1267.771741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1267.835838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1267.858751] LNet: 68370:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1267.868380] LNetError: 68370:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1267.872983] LNet: Removed LNI 192.168.203.57@tcp [ 1267.895242] LNet: Removed LNI 192.168.203.57@tcp1 [ 1267.923970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1267.988255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1268.038089] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1268.809228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1268.840799] LNet: Added LNI 192.168.203.57@tcp1 [8/256/0/180] [ 1269.526724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.57@tcp [ 1269.551254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1269.617980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1269.670714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1269.715315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1269.785757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1269.821236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1269.868719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.57@tcp [ 1269.892680] LNetError: 68643:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.57@tcp to recovery [ 1270.047749] LNet: There was an unexpected network error while writing to 192.168.203.57: rc = -22 [ 1272.315789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1276.192197] LNet: There was an unexpected network error while writing to 192.168.203.57: rc = -22 [ 1276.204965] LNet: Skipped 1 previous similar message [ 1280.567514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1280.609855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1280.649175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1288.331252] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 15:20:49 (1787340049) [ 1288.884663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1288.932921] LNet: 69353:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1288.946437] LNetError: 69353:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1288.951335] LNetError: 69353:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1288.963394] LNet: Removed LNI 192.168.203.57@tcp [ 1288.990625] LNet: Removed LNI 192.168.203.57@tcp1 [ 1289.021158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1289.166963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1289.200703] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1290.078795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1290.135762] LNet: Added LNI 192.168.203.57@tcp1 [8/256/0/180] [ 1291.385093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.57@tcp [ 1291.485245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1291.564242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1291.648377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1291.733672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1291.791164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1291.839393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.57@tcp --health 0 [ 1292.037396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1292.192244] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.57@tcp [ 1292.198015] LNet: There was an unexpected network error while writing to 192.168.203.57: rc = -22 [ 1292.198024] LNet: Skipped 1 previous similar message [ 1307.507161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1307.652373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1307.691486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1307.727588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.57@tcp --health 500 [ 1307.854156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1314.582637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1314.670149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1314.696656] LNet: 69845:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1314.708815] LNetError: 69845:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1314.716773] LNetError: 69845:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1314.728029] LNet: Removed LNI 192.168.203.57@tcp [ 1314.744884] LNet: Removed LNI 192.168.203.57@tcp1 [ 1314.769457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1314.832736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1314.852108] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1315.497730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1315.524566] LNet: Added LNI 192.168.203.57@tcp1 [8/256/0/180] [ 1316.331842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.57@tcp [ 1316.378530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1316.482115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1316.520648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1316.564682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1316.626944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1316.685969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1316.729339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.57@tcp --health 0 [ 1317.855706] LNet: There was an unexpected network error while writing to 192.168.203.57: rc = -22 [ 1317.863139] LNet: Skipped 3 previous similar messages [ 1318.040839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1328.225121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1328.290209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1328.337674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1334.238077] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 15:21:35 (1787340095) [ 1334.743751] LNet: 70891:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1334.751788] LNetError: 70891:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1334.759288] LNetError: 70891:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1334.768776] LNet: Removed LNI 192.168.203.57@tcp [ 1334.784461] LNet: Removed LNI 192.168.203.57@tcp1 [ 1335.362631] Key type .llcrypt unregistered [ 1335.366628] Key type ._llcrypt unregistered [ 1342.531815] Key type ._llcrypt registered [ 1342.535440] Key type .llcrypt registered [ 1342.856909] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1342.901306] alg: No test for adler32 (adler32-zlib) [ 1343.881611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1343.903563] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1354.349543] Lustre: DEBUG MARKER: Initial discovery [ 1354.412374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.157@tcp [ 1356.804875] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1356.909469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1356.979914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.157@tcp [ 1357.028331] LNet: There was an unexpected network error while writing to 192.168.203.157: rc = -22 [ 1357.051575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1357.101976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.203.157@tcp -r 1 -m GET -l 3 [ 1357.136149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.203.157@tcp -r 1 -m GET -e local_error [ 1357.179344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.203.157@tcp -r 1 -m PUT -l 6 [ 1357.215380] LNet: 1 local NIs in recovery (showing 1): 192.168.203.57@tcp [ 1358.664528] Lustre: DEBUG MARKER: Force oleg357-client.virtnet to discover 192.168.203.157@tcp (in background) [ 1358.739071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.157@tcp [ 1360.218726] Lustre: DEBUG MARKER: Force oleg357-server to discover 192.168.203.57@tcp [ 1361.825553] LNet: There was an unexpected network error while writing to 192.168.203.157: rc = -22 [ 1362.740335] Lustre: DEBUG MARKER: Wait for 71651 [ 1364.815715] Lustre: DEBUG MARKER: Finished wait on 71651 [ 1364.861494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1364.946291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1364.952246] LNet: There was an unexpected network error while writing to 192.168.203.157: rc = -22 [ 1365.281107] LNet: 72112:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1365.291380] LNetError: 72112:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1366.310421] LNet: Removed LNI 192.168.203.57@tcp [ 1366.795513] Key type .llcrypt unregistered [ 1366.800260] Key type ._llcrypt unregistered [ 1382.817094] Key type ._llcrypt registered [ 1382.820334] Key type .llcrypt registered [ 1384.334697] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 15:22:24 (1787340144) [ 1385.043608] Key type .llcrypt unregistered [ 1385.049678] Key type ._llcrypt unregistered [ 1391.008647] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1391.273264] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1391.280430] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1391.291910] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1393.913812] Key type ._llcrypt registered [ 1393.918652] Key type .llcrypt registered [ 1394.157237] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1394.184552] alg: No test for adler32 (adler32-zlib) [ 1395.141569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1395.220369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1395.252725] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1396.126847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1396.159972] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1404.066531] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 15:22:45 (1787340165) [ 1404.142953] LNet: 74184:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1404.151994] LNetError: 74184:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1404.160418] LNet: Removed LNI 10.1.2.3@tcp [ 1404.172333] LNet: Removed LNI 192.168.203.57@tcp [ 1404.698664] Key type .llcrypt unregistered [ 1404.705062] Key type ._llcrypt unregistered [ 1410.660848] Key type ._llcrypt registered [ 1410.664209] Key type .llcrypt registered [ 1410.906994] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1410.931901] alg: No test for adler32 (adler32-zlib) [ 1411.876988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1411.931429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1411.949391] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1412.609307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1412.624466] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1413.594611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.203.57@tcp [ 1414.688055] LNetError: Unexpected error -22 connecting to 192.168.203.57@tcp at host 192.168.203.57:988 [ 1420.226275] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 15:23:01 (1787340181) [ 1420.484412] LNet: 75328:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1420.496059] LNetError: 75328:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1420.505314] LNet: Removed LNI 10.1.2.3@tcp [ 1420.525438] LNet: Removed LNI 192.168.203.57@tcp [ 1421.321629] Key type .llcrypt unregistered [ 1421.327985] Key type ._llcrypt unregistered [ 1428.413883] Key type ._llcrypt registered [ 1428.423376] Key type .llcrypt registered [ 1428.817211] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1428.847554] alg: No test for adler32 (adler32-zlib) [ 1429.919836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1429.997282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1430.059951] LNet: Added LNI 192.168.203.57@tcp1 [8/256/0/180] [ 1430.934779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1430.963100] LNet: Added LNI 192.168.203.57@tcp2 [8/256/0/180] [ 1431.637314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.203.57@tcp1 --nid 192.168.203.57@tcp2 [ 1436.282598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.57@tcp1 192.168.203.57@tcp2 [ 1436.309801] LNetError: 76169:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.203.57@tcp1 to 192.168.203.57@tcp2 - Route table may be misconfigured [ 1436.346154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.57@tcp2 192.168.203.57@tcp1 [ 1443.023416] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 15:23:23 (1787340203) [ 1443.129963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1443.159198] LNet: 76742:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1443.165987] LNetError: 76742:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1443.171408] LNet: Removed LNI 192.168.203.57@tcp1 [ 1444.201517] LNet: Removed LNI 192.168.203.57@tcp2 [ 1444.243571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1444.277527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1444.298316] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1445.114111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1445.152035] LNet: Added LNI 192.168.203.57@tcp1 [8/256/0/180] [ 1446.104377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.57@tcp [ 1446.142816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.57@tcp 192.168.203.57@tcp [ 1446.195876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.57@tcp1 192.168.203.57@tcp1 [ 1446.255090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.57@tcp -d 192.168.203.57@tcp -e network_timeout [ 1446.317237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.57@tcp -d 192.168.203.57@tcp1 -e network_timeout [ 1446.376950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp -e network_timeout [ 1446.468036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp1 -e network_timeout [ 1446.512483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.57@tcp [ 1446.534795] LNet: There was an unexpected network error while writing to 192.168.203.57: rc = -22 [ 1446.550130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1446.625949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1446.731632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1447.328123] LNet: 1 local NIs in recovery (showing 1): 192.168.203.57@tcp [ 1452.417580] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 15:23:33 (1787340213) [ 1452.517326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1452.554214] LNet: 77601:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1452.584150] LNetError: 77601:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1452.588405] LNetError: 77601:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1452.599050] LNet: Removed LNI 192.168.203.57@tcp [ 1453.681117] LNet: Removed LNI 192.168.203.57@tcp1 [ 1453.741946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1453.851891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1454.222333] LNet: 77647:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1454.669524] Key type .llcrypt unregistered [ 1454.675733] Key type ._llcrypt unregistered [ 1464.948681] Key type ._llcrypt registered [ 1464.952098] Key type .llcrypt registered [ 1466.154650] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 15:23:46 (1787340226) [ 1466.986238] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1467.016827] alg: No test for adler32 (adler32-zlib) [ 1468.065086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1469.931460] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1471.759176] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 15:23:52 (1787340232) [ 1471.884745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1471.935949] LNet: 78740:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1471.976255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1472.077195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1472.137193] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1473.122927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1473.152866] LNet: Added LNI 192.168.203.57@tcp1 [8/256/0/180] [ 1474.318300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.57@tcp [ 1474.364105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.57@tcp1 [ 1474.442932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.57@tcp1 [ 1482.054421] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 15:24:02 (1787340242) [ 1483.638601] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg357-server [ 1485.299761] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 15:24:06 (1787340246) [ 1486.847885] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg357-server [ 1488.475403] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 15:24:09 (1787340249) [ 1489.867214] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg357-server [ 1491.589909] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 15:24:12 (1787340252) [ 1493.432830] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg357-server [ 1494.858489] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 15:24:15 (1787340255) [ 1496.971877] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg357-server [ 1498.806150] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 15:24:19 (1787340259) [ 1500.571073] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg357-server [ 1502.385329] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 15:24:23 (1787340263) [ 1504.427330] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg357-server [ 1506.320549] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 15:24:26 (1787340266) [ 1508.108668] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg357-server [ 1509.589674] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 15:24:30 (1787340270) [ 1511.338194] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg357-server [ 1513.056239] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 15:24:33 (1787340273) [ 1513.400986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1513.424522] LNet: 82631:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1513.431236] LNetError: 82631:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1513.439910] LNet: Removed LNI 192.168.203.57@tcp [ 1514.473437] LNet: Removed LNI 192.168.203.57@tcp1 [ 1514.519299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1514.596285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1514.621789] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1515.246107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1515.358983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.57@tcp [ 1515.641164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1515.668819] LNet: 82778:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1515.680709] LNetError: 82778:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1515.691500] LNetError: 82778:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1515.708415] LNet: Removed LNI 192.168.203.57@tcp [ 1515.744718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1515.814645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1515.844157] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1516.732699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.57@tcp [ 1517.028911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1517.102696] LNetError: 82923:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1517.126932] LNet: Removed LNI 192.168.203.57@tcp [ 1517.168676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1517.241527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1517.275435] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1518.261585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1518.336299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.57@tcp [ 1518.517098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1518.543843] LNet: 83070:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1518.565262] LNet: 83070:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1519.654929] LNet: Removed LNI 192.168.203.57@tcp [ 1519.690582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1519.773555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1519.805221] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1520.666341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.57@tcp [ 1520.885392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1520.912613] LNetError: 83216:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1520.918511] LNetError: 83216:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1520.928982] LNet: Removed LNI 192.168.203.57@tcp [ 1520.956966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1521.018981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1521.038163] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1521.700503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1521.797179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.57@tcp [ 1522.029442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1523.121262] LNet: Removed LNI 192.168.203.57@tcp [ 1523.146609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1523.190515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1523.210314] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1523.888828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.57@tcp [ 1524.130194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1524.156554] LNet: 83509:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1524.162051] LNet: 83509:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1525.224293] LNet: Removed LNI 192.168.203.57@tcp [ 1525.249052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1525.302565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1525.322907] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1526.115997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1526.215758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.57@tcp [ 1526.539155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1526.571937] LNetError: 83656:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1526.580635] LNetError: 83656:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1526.599868] LNet: Removed LNI 192.168.203.57@tcp [ 1526.629089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1526.717872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1526.753110] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1527.530197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.57@tcp [ 1527.732874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1528.821893] LNet: Removed LNI 192.168.203.57@tcp [ 1528.863368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1528.942631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1528.974929] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1529.758336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1529.838409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.57@tcp [ 1530.114293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1531.179165] LNet: Removed LNI 192.168.203.57@tcp [ 1531.214324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1531.283171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1531.324456] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1531.934654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.57@tcp [ 1532.145057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1532.179717] LNet: 84094:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1532.190431] LNet: 84094:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1533.229602] LNet: Removed LNI 192.168.203.57@tcp [ 1533.257254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1533.303848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1533.328341] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1533.924738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1534.000309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.57@tcp [ 1534.263494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1535.336341] LNet: Removed LNI 192.168.203.57@tcp [ 1535.381683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1535.459457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1535.490963] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1536.196378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.57@tcp [ 1537.808450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1537.843593] LNetError: 84396:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1537.849867] LNetError: 84396:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1537.881439] LNet: Removed LNI 192.168.203.57@tcp [ 1537.904301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1537.963137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1537.991204] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1538.988161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1539.036696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1539.070790] LNet: Removed LNI 192.168.203.57@tcp [ 1539.103317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1539.163819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1539.186043] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1540.133632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1540.194408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1540.221794] LNet: Removed LNI 192.168.203.57@tcp [ 1540.243643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1540.315463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1540.450979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1540.510113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1540.576909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1540.648549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1540.672600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1540.708803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1540.769287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1540.792319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1540.837320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1540.865754] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1541.940485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1541.970915] LNet: Removed LNI 192.168.203.57@tcp [ 1541.986176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1542.044129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1550.235422] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 15:25:10 (1787340310) [ 1550.527065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1550.567154] LNet: 85418:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1550.577284] LNet: 85418:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 1550.610835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1550.742180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1550.778817] LNet: Added LNI 192.168.203.57@tcp231 [8/256/0/180] [ 1552.113563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1552.142968] LNet: Removed LNI 192.168.203.57@tcp231 [ 1552.160665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1552.274797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1552.316830] LNet: Added LNI 192.168.203.57@tcp231 [8/256/0/180] [ 1552.321194] LNet: 85564:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1552.826914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1552.881856] LNet: Removed LNI 192.168.203.57@tcp231 [ 1552.909098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1552.936792] LNet: Added LNI 192.168.203.57@tcp231 [8/256/0/180] [ 1554.364666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1554.395388] LNetError: 85708:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1554.404893] LNetError: 85708:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1554.426655] LNet: Removed LNI 192.168.203.57@tcp231 [ 1554.440802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1554.526299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1554.588592] LNet: Added LNI 192.168.203.57@tcp231 [8/256/0/0] [ 1554.599593] LNet: 85716:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1555.185266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1555.224983] LNet: Removed LNI 192.168.203.57@tcp231 [ 1555.249734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1555.296721] LNet: Added LNI 192.168.203.57@tcp231 [8/256/0/0] [ 1556.501149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1556.523755] LNet: Removed LNI 192.168.203.57@tcp231 [ 1556.543965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1556.596178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1556.615458] LNet: Added LNI 192.168.203.57@tcp231 [8/256/0/60] [ 1556.618934] LNet: 85868:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1557.098782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1557.145056] LNet: Removed LNI 192.168.203.57@tcp231 [ 1557.170246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1557.196441] LNet: Added LNI 192.168.203.57@tcp231 [8/256/0/60] [ 1558.420414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1558.450699] LNet: Removed LNI 192.168.203.57@tcp231 [ 1558.472813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1558.534478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1558.554965] LNet: Added LNI 192.168.203.57@tcp231 [8/256/0/180] [ 1559.070733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1559.116857] LNet: Removed LNI 192.168.203.57@tcp231 [ 1559.143331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1559.174949] LNet: Added LNI 192.168.203.57@tcp231 [8/256/0/180] [ 1567.933912] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 15:25:28 (1787340328) [ 1568.079157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1568.149597] LNet: Removed LNI 192.168.203.57@tcp231 [ 1568.205273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1568.270463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1568.307380] LNet: Added LNI 192.168.203.57@tcp232 [8/256/0/180] [ 1568.915508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1568.955747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1568.988897] LNet: Removed LNI 192.168.203.57@tcp232 [ 1569.022180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1569.048768] LNet: Added LNI 192.168.203.57@tcp232 [8/256/0/180] [ 1569.625310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1569.657587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1569.683805] LNet: Removed LNI 192.168.203.57@tcp232 [ 1569.697235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1569.719044] LNet: Added LNI 192.168.203.57@tcp232 [8/256/0/180] [ 1570.339829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1570.386933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1570.424898] LNet: Removed LNI 192.168.203.57@tcp232 [ 1570.449119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1570.473296] LNet: Added LNI 192.168.203.57@tcp232 [8/256/0/180] [ 1571.195563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1571.267076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1571.317761] LNet: Removed LNI 192.168.203.57@tcp232 [ 1571.336267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1577.545830] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 15:25:38 (1787340338) [ 1578.758669] Key type .llcrypt unregistered [ 1578.764151] Key type ._llcrypt unregistered [ 1585.111773] Key type ._llcrypt registered [ 1585.114784] Key type .llcrypt registered [ 1585.298850] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1585.321409] alg: No test for adler32 (adler32-zlib) [ 1586.228780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1586.278135] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1594.106577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.157@tcp [ 1603.584459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1603.658977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1603.699932] LNet: Added LNI 192.168.203.57@tcp1 [8/256/0/180] [ 1605.410789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp -m GET -e local_interrupt -r 2 [ 1605.447811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp1 -m GET -e local_interrupt -r 2 [ 1605.488526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp -m GET -e local_interrupt -r 2 [ 1605.542470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp1 -m GET -e local_interrupt -r 2 [ 1605.583189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp -m GET -e local_interrupt -r 2 [ 1605.650295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp1 -m GET -e local_interrupt -r 2 [ 1605.714914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp -m GET -e local_interrupt -r 2 [ 1605.765668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp1 -m GET -e local_interrupt -r 2 [ 1605.832281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.57@tcp 192.168.203.157@tcp [ 1605.873628] LNet: There was an unexpected network error while writing to 192.168.203.157: rc = -22 [ 1605.878361] LNet: 1 local NIs in recovery (showing 1): 192.168.203.57@tcp [ 1606.171494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1606.196368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1606.235316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1606.278742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1606.330161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1606.392633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1606.429307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1606.530857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1606.559390] LNet: 88525:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1606.591715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1606.665577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1606.735622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1606.778935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1606.831461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1606.878536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1607.485686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp -m GET -e local_dropped -r 2 [ 1607.529922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp1 -m GET -e local_dropped -r 2 [ 1607.588882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp -m GET -e local_dropped -r 2 [ 1607.648872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp1 -m GET -e local_dropped -r 2 [ 1607.722992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp -m GET -e local_dropped -r 2 [ 1607.802741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp1 -m GET -e local_dropped -r 2 [ 1607.880667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp -m GET -e local_dropped -r 2 [ 1607.944399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp1 -m GET -e local_dropped -r 2 [ 1608.014806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.57@tcp 192.168.203.157@tcp [ 1608.062532] LNet: There was an unexpected network error while writing to 192.168.203.157: rc = -22 [ 1608.399515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1608.458484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1608.490923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1608.531130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1608.580914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1608.621339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1608.661374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1608.714876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1608.738478] LNet: 88645:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1608.765260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1608.812351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1608.869994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1608.908525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1608.937157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1608.969749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1609.657975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp -m GET -e local_aborted -r 2 [ 1609.721612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp1 -m GET -e local_aborted -r 2 [ 1609.760535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp -m GET -e local_aborted -r 2 [ 1609.808924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp1 -m GET -e local_aborted -r 2 [ 1609.866433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp -m GET -e local_aborted -r 2 [ 1609.944810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp1 -m GET -e local_aborted -r 2 [ 1609.984319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp -m GET -e local_aborted -r 2 [ 1610.039826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp1 -m GET -e local_aborted -r 2 [ 1610.101825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.57@tcp 192.168.203.157@tcp [ 1610.136831] LNet: There was an unexpected network error while writing to 192.168.203.157: rc = -22 [ 1610.424890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1610.477564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1610.547616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1610.610035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1610.664695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1610.713889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1610.760360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1610.792945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1610.816686] LNet: 88766:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1610.844560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1610.880963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1610.930224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1610.985121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1611.045424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1611.086870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1611.642631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp -m GET -e local_no_route -r 2 [ 1611.704931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp1 -m GET -e local_no_route -r 2 [ 1611.755147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp -m GET -e local_no_route -r 2 [ 1611.809464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp1 -m GET -e local_no_route -r 2 [ 1611.858428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp -m GET -e local_no_route -r 2 [ 1611.901520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp1 -m GET -e local_no_route -r 2 [ 1611.955544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp -m GET -e local_no_route -r 2 [ 1612.002959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp1 -m GET -e local_no_route -r 2 [ 1612.059583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.57@tcp 192.168.203.157@tcp [ 1612.369210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1612.397502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1612.427304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1612.471783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1612.504341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1612.543159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1612.589211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1612.625369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1612.677160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1612.723825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1612.761695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1612.808987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1612.837908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1612.889473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1613.422601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp -m GET -e local_timeout -r 2 [ 1613.474509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp1 -m GET -e local_timeout -r 2 [ 1613.519448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp -m GET -e local_timeout -r 2 [ 1613.568366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp1 -m GET -e local_timeout -r 2 [ 1613.643493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp -m GET -e local_timeout -r 2 [ 1613.708314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp1 -m GET -e local_timeout -r 2 [ 1613.760236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp -m GET -e local_timeout -r 2 [ 1613.801731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp1 -m GET -e local_timeout -r 2 [ 1613.859969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.57@tcp 192.168.203.157@tcp [ 1613.882808] LNet: There was an unexpected network error while writing to 192.168.203.157: rc = -22 [ 1613.887234] LNet: Skipped 1 previous similar message [ 1614.147863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1614.182315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1614.218918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1614.261335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1614.290103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1614.318852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1614.360843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1614.392398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1614.410501] LNet: 89006:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1614.415175] LNet: 89006:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1614.431525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1614.456367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1614.480570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1614.508133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1614.564474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1614.605700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1615.175400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp -m GET -e remote_dropped -r 2 [ 1615.217178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp1 -m GET -e remote_dropped -r 2 [ 1615.249770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp -m GET -e remote_dropped -r 2 [ 1615.296755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp1 -m GET -e remote_dropped -r 2 [ 1615.329679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp -m GET -e remote_dropped -r 2 [ 1615.366505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp1 -m GET -e remote_dropped -r 2 [ 1615.396931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp -m GET -e remote_dropped -r 2 [ 1615.429807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp1 -m GET -e remote_dropped -r 2 [ 1615.458734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.57@tcp 192.168.203.157@tcp [ 1615.481133] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.157@tcp [ 1615.683335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1615.724288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1615.776841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1615.811621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1615.845039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1615.878666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1615.921487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1615.946664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1615.984803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1616.042656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1616.084819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1616.141654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1616.188646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1616.240523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1621.967864] LNet: 89205:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1621.977954] LNetError: 89205:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1621.994199] LNet: Removed LNI 192.168.203.57@tcp [ 1622.008636] LNet: Removed LNI 192.168.203.57@tcp1 [ 1622.335648] Key type .llcrypt unregistered [ 1622.337940] Key type ._llcrypt unregistered [ 1638.281827] Key type ._llcrypt registered [ 1638.288110] Key type .llcrypt registered [ 1639.634283] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 15:26:40 (1787340400) [ 1640.117894] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1640.138705] alg: No test for adler32 (adler32-zlib) [ 1641.037658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1641.119653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1641.145744] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1641.960712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1642.000842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.57@tcp [ 1642.062119] LNetError: 90186:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.57@tcp to recovery [ 1642.143326] LNet: 1 local NIs in recovery (showing 1): 192.168.203.57@tcp [ 1642.160809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.203.57@tcp [ 1642.245846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1648.440843] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 15:26:49 (1787340409) [ 1648.593493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1648.639920] LNet: 90775:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1648.648559] LNetError: 90775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1648.665545] LNet: Removed LNI 192.168.203.57@tcp [ 1648.681596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1648.748548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1648.771976] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1649.482617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1649.521661] LNet: Added LNI 192.168.203.57@tcp1 [8/256/0/180] [ 1650.525776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.203.57@tcp [ 1650.559537] LNetError: 91041:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.57@tcp to recovery [ 1650.787360] LNet: There was an unexpected network error while writing to 192.168.203.57: rc = -22 [ 1651.788201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.203.57@tcp [ 1658.638773] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 15:26:59 (1787340419) [ 1660.290661] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg357-server [ 1661.914310] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 15:27:02 (1787340422) [ 1663.885204] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg357-server [ 1665.322534] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 15:27:06 (1787340426) [ 1666.829758] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg357-server [ 1668.540775] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 15:27:09 (1787340429) [ 1668.781137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1668.795832] LNet: 92664:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1668.805605] LNetError: 92664:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1668.811517] LNet: Removed LNI 192.168.203.57@tcp [ 1668.821045] LNet: Removed LNI 192.168.203.57@tcp1 [ 1668.834411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1668.893481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1668.920121] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1669.531309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1669.599424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1669.651741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1669.707648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1669.747573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1669.788091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1675.775498] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 15:27:16 (1787340436) [ 1676.037661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1676.087400] LNet: 93389:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1676.097181] LNetError: 93389:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1676.106908] LNetError: 93389:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1676.136871] LNet: Removed LNI 192.168.203.57@tcp [ 1676.166672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1676.230839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1676.272356] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1682.932814] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 15:27:23 (1787340443) [ 1683.020334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1683.060669] LNet: 94085:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1683.067207] LNetError: 94085:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1683.090453] LNet: Removed LNI 192.168.203.57@tcp [ 1683.141542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1683.250614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1683.290459] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1690.487736] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 15:27:31 (1787340451) [ 1692.065295] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1693.818499] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 15:27:34 (1787340454) [ 1694.346656] LNet: 95126:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1694.368115] LNetError: 95126:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1694.386973] LNet: Removed LNI 192.168.203.57@tcp [ 1695.087919] Key type .llcrypt unregistered [ 1695.094506] Key type ._llcrypt unregistered [ 1701.960204] Key type ._llcrypt registered [ 1701.966927] Key type .llcrypt registered [ 1702.169957] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1702.189386] alg: No test for adler32 (adler32-zlib) [ 1703.080660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1703.115276] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1710.908338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.157@tcp [ 1716.040512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.203.157@tcp [ 1716.495572] LNet: 95551:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1716.512952] LNetError: 95551:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1716.536192] LNet: Removed LNI 192.168.203.57@tcp [ 1716.999585] Key type .llcrypt unregistered [ 1717.002413] Key type ._llcrypt unregistered [ 1728.775439] Key type ._llcrypt registered [ 1728.779329] Key type .llcrypt registered [ 1730.503054] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1732.286532] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1734.109122] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 15:28:14 (1787340494) [ 1735.971624] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1737.686879] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 15:28:18 (1787340498) [ 1739.526460] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg357-server [ 1741.062712] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 15:28:22 (1787340502) [ 1743.349984] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg357-server [ 1745.423537] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 15:28:25 (1787340505) [ 1747.299184] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1747.326890] alg: No test for adler32 (adler32-zlib) [ 1748.391781] Lustre: Lustre: Build Version: 2.17.57_44_g85e5bce [ 1748.739361] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1750.527160] Key type lgssc registered [ 1752.076857] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1776.141552] Key type lgssc unregistered [ 1776.534449] LNet: 98728:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1776.548482] LNetError: 98728:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1776.568769] LNet: Removed LNI 192.168.203.57@tcp [ 1777.420678] Key type .llcrypt unregistered [ 1777.428970] Key type ._llcrypt unregistered [ 1792.690209] Key type ._llcrypt registered [ 1792.692827] Key type .llcrypt registered [ 1794.140592] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 15:29:14 (1787340554) [ 1794.703425] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1794.714755] alg: No test for adler32 (adler32-zlib) [ 1795.664213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1795.733671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1795.784698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1795.823337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1795.872132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1803.478635] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 15:29:23 (1787340563) [ 1803.626130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1803.660346] LNet: 100137:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1803.715227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1803.808421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1803.874658] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1804.740453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1804.761902] LNet: 100288:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1804.770430] LNetError: 100288:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1804.781795] LNet: Removed LNI 192.168.203.57@tcp [ 1804.794630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1804.830526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1804.849836] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1805.351684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1805.374829] LNetError: 100317:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1805.403514] LNet: Removed LNI 192.168.203.57@tcp [ 1805.427595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1805.496236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1805.543329] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1813.385951] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 15:29:33 (1787340573) [ 1813.618430] LNet: 100905:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1813.627966] LNet: 100905:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1813.640758] LNetError: 100905:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1813.667912] LNet: Removed LNI 192.168.203.57@tcp [ 1814.265990] Key type .llcrypt unregistered [ 1814.270485] Key type ._llcrypt unregistered [ 1820.921622] Key type ._llcrypt registered [ 1820.927486] Key type .llcrypt registered [ 1821.153825] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1821.169374] alg: No test for adler32 (adler32-zlib) [ 1822.105211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1822.227792] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1822.230172] LNet: *** cfs_fail_loc=e003, val=100*** [ 1822.921957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.57@tcp [ 1822.954438] LNet: *** cfs_fail_loc=e003, val=100*** [ 1822.960356] LNet: Skipped 2 previous similar messages [ 1822.970446] Ping reply received with mapping: [ 1822.974855] found metadata reply NI[192.168.203.57@tcp] [ 1822.979530] 39 cb a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1822.982802] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1822.987444] 5a 5a 5a 5a 5a 5a 5a 5a [ 1823.165387] LNet: 101308:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1823.195689] LNetError: 101308:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1823.217815] LNet: Removed LNI 192.168.203.57@tcp [ 1823.956597] Key type .llcrypt unregistered [ 1823.962945] Key type ._llcrypt unregistered [ 1836.456536] Key type ._llcrypt registered [ 1836.459589] Key type .llcrypt registered [ 1837.850486] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 15:29:58 (1787340598) [ 1838.396201] Key type .llcrypt unregistered [ 1838.405380] Key type ._llcrypt unregistered [ 1844.837837] Key type ._llcrypt registered [ 1844.841240] Key type .llcrypt registered [ 1845.108438] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1845.127588] alg: No test for adler32 (adler32-zlib) [ 1846.294337] LNetError: 102365:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1846.748804] Key type .llcrypt unregistered [ 1846.754763] Key type ._llcrypt unregistered [ 1851.588793] Key type ._llcrypt registered [ 1851.592158] Key type .llcrypt registered [ 1853.301572] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 15:30:13 (1787340613) [ 1854.019384] Key type .llcrypt unregistered [ 1854.023071] Key type ._llcrypt unregistered [ 1860.446082] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1860.450198] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1863.106798] Key type ._llcrypt registered [ 1863.109654] Key type .llcrypt registered [ 1863.348724] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1863.393719] alg: No test for adler32 (adler32-zlib) [ 1864.368387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1864.445459] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1873.677527] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 15:30:34 (1787340634) [ 1873.851076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1873.878583] LNet: 104155:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1873.886429] LNetError: 104155:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1873.904198] LNet: Removed LNI 10.1.2.3@tcp [ 1873.927457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1874.568625] Key type .llcrypt unregistered [ 1874.572740] Key type ._llcrypt unregistered [ 1880.255648] Key type ._llcrypt registered [ 1880.260658] Key type .llcrypt registered [ 1880.523515] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1880.548504] alg: No test for adler32 (adler32-zlib) [ 1881.500104] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1881.566815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.57@tcp [ 1881.769930] LNet: 104447:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1881.778260] LNetError: 104447:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1881.804595] LNet: Removed LNI 192.168.203.57@tcp [ 1882.288544] Key type .llcrypt unregistered [ 1882.292352] Key type ._llcrypt unregistered [ 1894.559549] Key type ._llcrypt registered [ 1894.563913] Key type .llcrypt registered [ 1896.565928] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 15:30:56 (1787340656) [ 1897.954308] Key type .llcrypt unregistered [ 1897.957597] Key type ._llcrypt unregistered [ 1905.559915] Key type ._llcrypt registered [ 1905.566812] Key type .llcrypt registered [ 1905.876647] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1905.903898] alg: No test for adler32 (adler32-zlib) [ 1914.771802] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 15:31:15 (1787340675) [ 1915.184667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1915.233721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1915.336279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1915.380686] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1916.330451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1916.379854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1916.418884] LNet: Removed LNI 192.168.203.57@tcp [ 1916.756316] LNet: 106203:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1917.198482] Key type .llcrypt unregistered [ 1917.200231] Key type ._llcrypt unregistered [ 1931.729908] Key type ._llcrypt registered [ 1931.732694] Key type .llcrypt registered [ 1933.700736] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 15:31:33 (1787340693) [ 1934.198547] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1934.221849] alg: No test for adler32 (adler32-zlib) [ 1935.311420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1935.422928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1935.479802] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1936.748633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.57@tcp [ 1936.842737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.203.57@tcp [ 1936.882310] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1936.889712] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1936.897498] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1936, 0, 0 [ 1936.900849] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1936.904870] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1936.909046] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1936.912542] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1936, 0, 0 [ 1936.919043] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1936.927623] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1936.929892] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1936.933580] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1936, 0, 1986 [ 1936.939126] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1936.942872] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1936.947800] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1936.953584] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1936, 0, 1986 [ 1936.957712] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1944.643775] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 15:31:45 (1787340705) [ 1945.267970] LNet: 107744:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1945.283959] LNetError: 107744:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1945.300428] LNet: Removed LNI 192.168.203.57@tcp [ 1946.126686] Key type .llcrypt unregistered [ 1946.131111] Key type ._llcrypt unregistered [ 1952.611286] Key type ._llcrypt registered [ 1952.613882] Key type .llcrypt registered [ 1952.805662] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1952.815963] alg: No test for adler32 (adler32-zlib) [ 1953.718618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1953.765539] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1962.185514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.157@tcp [ 1972.722491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1972.823275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1972.856574] LNet: Added LNI 192.168.203.57@tcp1 [8/256/0/180] [ 1974.182732] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1974.346538] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1974.360178] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1974.370182] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1976.802825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1976.824265] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1988.145079] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 15:32:28 (1787340748) [ 1988.483355] LNet: 109257:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1988.489572] LNetError: 109257:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1988.501393] LNet: Removed LNI 192.168.203.57@tcp [ 1989.536295] LNet: Removed LNI 192.168.203.57@tcp1 [ 1989.549327] LNetError: 109257:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1989.555405] LNetError: 109257:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1989.572705] LNet: Removed LNI 10.1.2.3@tcp99 [ 1990.309606] Key type .llcrypt unregistered [ 1990.313313] Key type ._llcrypt unregistered [ 1996.184692] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1996.197290] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1998.710157] Key type ._llcrypt registered [ 1998.714366] Key type .llcrypt registered [ 1998.889829] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1998.906626] alg: No test for adler32 (adler32-zlib) [ 1999.846154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1999.910582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1999.944276] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 2000.626892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2000.658152] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2002.454857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2002.503375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2002.537442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2002.558920] LNetError: 110005:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2002.599341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2002.638328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.57@tcp [ 2003.190696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2003.258795] LNet: 110025:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2003.265476] LNetError: 110025:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2003.269837] LNet: Removed LNI 10.1.2.3@tcp [ 2004.335552] LNet: Removed LNI 192.168.203.57@tcp [ 2004.982547] Key type .llcrypt unregistered [ 2004.986381] Key type ._llcrypt unregistered [ 2017.288172] Key type ._llcrypt registered [ 2017.290400] Key type .llcrypt registered [ 2018.980470] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 15:32:59 (1787340779) [ 2019.316025] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2019.333401] alg: No test for adler32 (adler32-zlib) [ 2020.215900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2020.261952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2020.282242] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 2021.018948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.57@tcp [ 2021.108645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg357-client@tcp [ 2027.489762] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 15:33:08 (1787340788) [ 2027.625742] LNet: 111553:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2027.646552] LNetError: 111553:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2027.674394] LNet: Removed LNI 192.168.203.57@tcp [ 2028.713868] Key type .llcrypt unregistered [ 2028.718982] Key type ._llcrypt unregistered [ 2035.265379] Key type ._llcrypt registered [ 2035.271158] Key type .llcrypt registered [ 2035.497151] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2035.508133] alg: No test for adler32 (adler32-zlib) [ 2036.451982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2036.491549] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 2036.516914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2043.144825] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 15:33:23 (1787340803) [ 2045.019374] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2047.119234] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 15:33:27 (1787340807) [ 2048.687673] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2050.132561] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 15:33:31 (1787340811) [ 2050.365607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2050.390230] LNet: 112985:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2050.399584] LNetError: 112985:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2050.419372] LNet: Removed LNI 192.168.203.57@tcp [ 2050.443104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2050.502455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2050.521729] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 2051.298753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2051.350786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2057.594554] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 15:33:38 (1787340818) [ 2057.888226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2057.914561] LNet: 113695:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2057.922781] LNetError: 113695:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2057.934675] LNet: Removed LNI 192.168.203.57@tcp [ 2057.985566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2058.010576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2058.034201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2058.914432] Key type .llcrypt unregistered [ 2058.920610] Key type ._llcrypt unregistered [ 2070.888458] Key type ._llcrypt registered [ 2070.892764] Key type .llcrypt registered [ 2072.561059] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 15:33:52 (1787340832) [ 2073.086952] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2073.114698] alg: No test for adler32 (adler32-zlib) [ 2074.113470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2074.173470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2074.211278] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 2075.127490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2075.219142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.57@tcp [ 2083.806762] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 15:34:04 (1787340844) [ 2084.025570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2084.048770] LNet: 115256:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2084.062241] LNetError: 115256:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2084.086996] LNet: Removed LNI 192.168.203.57@tcp [ 2084.113655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2084.229758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2084.279619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2091.159951] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 15:34:11 (1787340851) [ 2091.421049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2091.440534] LNet: 115836:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2091.465463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2091.561430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2092.011245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2092.041480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2092.100978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2092.539564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2092.567564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2092.647141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2092.676953] LNet: 115869:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2092.708978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2092.758835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2093.293173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2093.351861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2093.443257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2093.967318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2094.002619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2100.468618] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 15:34:21 (1787340861) [ 2100.564852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2100.614894] LNet: 116462:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2100.662626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2100.724463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2100.744412] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 2101.654517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2101.700241] LNet: Added LNI 192.168.203.57@tcp2 [8/256/0/180] [ 2102.736829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2102.791777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.57@tcp [ 2103.076730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2103.096671] LNetError: 116748:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2103.104591] LNet: Removed LNI 192.168.203.57@tcp [ 2103.113811] LNet: Removed LNI 192.168.203.57@tcp2 [ 2103.138735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2103.218456] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 2103.224258] LNet: Added LNI 192.168.203.57@tcp2 [8/256/0/180] [ 2103.233418] LNet: 116754:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2103.709299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.57@tcp [ 2103.908747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2104.991255] LNet: Removed LNI 192.168.203.57@tcp [ 2104.996918] LNetError: 116792:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2105.001785] LNetError: 116792:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2105.020877] LNet: Removed LNI 192.168.203.57@tcp2 [ 2105.065975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2105.172995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2105.220245] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 2106.082879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2106.123134] LNet: Added LNI 192.168.203.57@tcp2 [8/256/0/180] [ 2107.149632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.57@tcp [ 2107.201406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2107.287464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.57@tcp [ 2107.466666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2107.489579] LNet: 117076:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2107.495436] LNet: 117076:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2107.503356] LNetError: 117076:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2107.509471] LNet: Removed LNI 192.168.203.57@tcp [ 2108.585667] LNet: Removed LNI 192.168.203.57@tcp2 [ 2108.620754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2108.705561] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 2108.713975] LNet: Added LNI 192.168.203.57@tcp2 [8/256/0/180] [ 2108.723421] LNet: 117082:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2109.134884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.57@tcp [ 2109.329370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2109.368452] LNet: Removed LNI 192.168.203.57@tcp [ 2110.441433] LNet: Removed LNI 192.168.203.57@tcp2 [ 2110.472911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2110.540221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2110.569885] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 2111.154856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2111.168582] LNet: Added LNI 192.168.203.57@tcp2 [8/256/0/180] [ 2111.863973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2111.926825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.57@tcp [ 2112.116493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2112.132617] LNetError: 117399:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2112.135693] LNetError: 117399:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2113.190302] LNet: Removed LNI 192.168.203.57@tcp [ 2113.204630] LNet: Removed LNI 192.168.203.57@tcp2 [ 2113.261668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2113.360191] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 2113.369437] LNet: Added LNI 192.168.203.57@tcp2 [8/256/0/180] [ 2113.376242] LNet: 117405:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2114.022558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.57@tcp [ 2114.259394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2115.295446] LNet: Removed LNI 192.168.203.57@tcp [ 2115.306363] LNet: Removed LNI 192.168.203.57@tcp2 [ 2115.343687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2115.413929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2115.438667] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 2116.152484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2116.171043] LNet: Added LNI 192.168.203.57@tcp2 [8/256/0/180] [ 2116.904280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.57@tcp [ 2116.938338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2117.017136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.57@tcp [ 2117.192112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2117.207142] LNet: 117726:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2117.214398] LNet: 117726:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2118.239220] LNet: Removed LNI 192.168.203.57@tcp [ 2119.272754] LNet: Removed LNI 192.168.203.57@tcp2 [ 2119.314108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2119.388642] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 2119.394804] LNet: Added LNI 192.168.203.57@tcp2 [8/256/0/180] [ 2119.400255] LNet: 117733:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2119.918973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.57@tcp [ 2120.137159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2120.174918] LNetError: 117770:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2120.182669] LNetError: 117770:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2120.200102] LNet: Removed LNI 192.168.203.57@tcp [ 2120.215681] LNet: Removed LNI 192.168.203.57@tcp2 [ 2120.236191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2120.288745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2120.323239] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 2120.897990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2120.918122] LNet: Added LNI 192.168.203.57@tcp2 [8/256/0/180] [ 2121.574786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2121.612328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2121.674859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.57@tcp [ 2121.952352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2122.975185] LNet: Removed LNI 192.168.203.57@tcp [ 2122.985419] LNet: Removed LNI 192.168.203.57@tcp2 [ 2123.017145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2123.079278] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 2123.085501] LNet: Added LNI 192.168.203.57@tcp2 [8/256/0/180] [ 2123.659880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.57@tcp [ 2123.991613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2125.087304] LNet: Removed LNI 192.168.203.57@tcp [ 2125.099994] LNet: Removed LNI 192.168.203.57@tcp2 [ 2125.140703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2125.219503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2125.252404] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 2125.916930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2125.932665] LNet: Added LNI 192.168.203.57@tcp2 [8/256/0/180] [ 2126.655808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.57@tcp [ 2126.699875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2126.739774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2126.794768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.57@tcp [ 2127.093599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2128.159215] LNet: Removed LNI 192.168.203.57@tcp [ 2129.192863] LNet: Removed LNI 192.168.203.57@tcp2 [ 2129.219951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2129.291632] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 2129.298187] LNet: Added LNI 192.168.203.57@tcp2 [8/256/0/180] [ 2129.302478] LNet: 118432:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2129.311143] LNet: 118432:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 2129.754741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.57@tcp [ 2135.231920] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2136.879542] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 15:34:57 (1787340897) [ 2137.101346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2137.122177] LNet: 119191:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2137.132319] LNet: 119191:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2137.144219] LNetError: 119191:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2137.156688] LNetError: 119191:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2137.164780] LNet: Removed LNI 192.168.203.57@tcp [ 2138.214488] LNet: Removed LNI 192.168.203.57@tcp2 [ 2138.249832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2138.577613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2138.596872] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 2139.262376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2139.296115] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2140.135313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2140.148258] LNet: Removed LNI 10.1.2.3@tcp [ 2140.158151] LNet: Removed LNI 192.168.203.57@tcp [ 2140.182124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2140.247975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2140.295251] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 2140.316762] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2140.841940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2140.879700] LNet: Removed LNI 192.168.203.57@tcp [ 2140.907159] LNet: Removed LNI 10.1.2.3@tcp [ 2147.038858] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 15:35:07 (1787340907) [ 2147.169688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2147.218162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2147.293406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2147.330640] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 2148.176745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2148.232088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.57@tcp [ 2148.294821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.57@tcp [ 2148.332538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.57@tcp [ 2148.371806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.57@tcp [ 2148.423784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.57@tcp [ 2148.486981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.57@tcp [ 2148.554905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.57@tcp [ 2148.608261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.57@tcp [ 2148.662980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.57@tcp [ 2148.710344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.57@tcp [ 2148.918672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2154.935889] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 15:35:15 (1787340915) [ 2155.348472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2156.456284] LNet: Removed LNI 192.168.203.57@tcp [ 2156.494683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2156.658130] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2156.881923] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2156.887383] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2156.891083] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2159.224496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2159.257953] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 2159.802747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2159.825830] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2161.581785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.57@tcp [ 2161.630996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2161.689542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.203.57@tcp -m PUT -e local_timeout -r 1 [ 2161.743994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2161.820468] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2161.832487] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2162.094438] LNetError: 120870:0:(lib-move.c:4801:lnet_parse()) 192.168.203.57@tcp, src 192.168.203.57@tcp: Dropping PUT (error -108 looking up sender) [ 2163.104748] LNet: Removed LNI 10.1.2.3@tcp [ 2164.140696] LNet: Removed LNI 192.168.203.57@tcp [ 2164.567925] Key type .llcrypt unregistered [ 2164.570993] Key type ._llcrypt unregistered [ 2174.946804] Key type ._llcrypt registered [ 2174.949158] Key type .llcrypt registered [ 2176.032627] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 15:35:36 (1787340936) [ 2176.405237] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2176.437762] alg: No test for adler32 (adler32-zlib) [ 2177.306854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2177.605690] LNet: 122186:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2178.155392] Key type .llcrypt unregistered [ 2178.157021] Key type ._llcrypt unregistered [ 2183.735080] Key type ._llcrypt registered [ 2183.741960] Key type .llcrypt registered [ 2184.109890] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2184.137071] alg: No test for adler32 (adler32-zlib) [ 2185.192254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2185.228574] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 2193.530258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.157@tcp [ 2193.752789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2194.427634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.157@tcp [ 2194.465517] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.157@tcp [ 2194.713928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2194.750826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2194.778524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2194.810215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2194.838655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2194.870835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2194.911671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2194.960207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2194.975423] LNet: 122639:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2195.003222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2195.031249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2195.060957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2195.087975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2195.117375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2195.157789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2195.821610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2196.044537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2196.078414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2196.108531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2196.164993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2196.199057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2196.251829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2196.299593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2196.347269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2196.370504] LNet: 122757:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2196.398938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2196.435479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2196.468831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2196.517238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2196.558859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2196.621073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2197.589725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.157@tcp [ 2197.913735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2197.957454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2198.033408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2198.095800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2198.126872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2198.161187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2198.195577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2198.258161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2198.282285] LNet: 122874:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2198.295404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2198.346704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2198.406493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2198.451802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2198.513226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2198.562462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2199.211069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2199.458494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2199.492796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2199.533156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2199.584154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2199.644425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2199.703452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2199.753349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2199.795739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2199.852458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2199.896246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2199.942247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2199.983538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2200.050262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2200.099262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2200.529497] LNet: 123041:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2200.540106] LNetError: 123041:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2200.560968] LNet: Removed LNI 192.168.203.57@tcp [ 2200.920415] Key type .llcrypt unregistered [ 2200.924124] Key type ._llcrypt unregistered [ 2218.068726] Key type ._llcrypt registered [ 2218.074952] Key type .llcrypt registered [ 2219.243051] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 15:36:20 (1787340980) [ 2219.861976] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2219.879131] alg: No test for adler32 (adler32-zlib) [ 2220.880139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2221.315597] LNet: 123929:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2222.015591] Key type .llcrypt unregistered [ 2222.017506] Key type ._llcrypt unregistered [ 2229.199806] Key type ._llcrypt registered [ 2229.216371] Key type .llcrypt registered [ 2229.442957] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2229.465969] alg: No test for adler32 (adler32-zlib) [ 2230.424978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2230.489173] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 2238.952508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.157@tcp [ 2248.361884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2248.433969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2248.496875] LNet: Added LNI 192.168.203.57@tcp1 [8/256/0/180] [ 2249.484464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2250.017274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.157@tcp [ 2250.194523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2250.218762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2250.243487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2250.289909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2250.346534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2250.392860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2250.432778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2250.486537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2250.522471] LNet: 124586:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2250.558310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2250.605000] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2250.637429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2250.665933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2250.708353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2250.732223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2251.306176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2251.552919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2251.583231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2251.619640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2251.660520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2251.691867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2251.726918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2251.761636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2251.786847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2251.805105] LNet: 124703:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2251.823902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2251.870359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2251.906529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2251.931906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2251.968164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2252.006682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2252.578397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.157@tcp [ 2252.827526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2252.853291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2252.883389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2252.919978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2252.958039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2253.015898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2253.043961] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.157@tcp [ 2253.077700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2253.136141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2253.161876] LNet: 124820:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2253.190730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2253.233692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2253.254786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2253.289184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2253.331569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2253.410551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2254.031239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2254.281920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2254.312363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2254.358672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2254.388794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2254.428731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2254.459822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2254.491771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2254.518607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2254.550935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2254.604599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2254.652871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2254.716508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2254.742988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2254.774674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2259.762663] LNet: 125022:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2259.772839] LNetError: 125022:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2259.785188] LNet: Removed LNI 192.168.203.57@tcp [ 2259.799451] LNet: Removed LNI 192.168.203.57@tcp1 [ 2260.247340] Key type .llcrypt unregistered [ 2260.250274] Key type ._llcrypt unregistered [ 2275.187559] Key type ._llcrypt registered [ 2275.191759] Key type .llcrypt registered [ 2276.469619] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 15:37:17 (1787341037) [ 2276.867174] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2276.881715] alg: No test for adler32 (adler32-zlib) [ 2277.798292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2277.887870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2277.934390] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 2278.539558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2278.575881] LNet: Added LNI 192.168.203.57@tcp2 [8/256/0/180] [ 2279.573270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.57@tcp [ 2279.722965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2280.802195] LNet: Removed LNI 192.168.203.57@tcp [ 2288.036310] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 15:37:28 (1787341048) [ 2289.634621] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2291.226868] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 15:37:32 (1787341052) [ 2298.499285] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 15:37:39 (1787341059) [ 2305.265721] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 15:37:46 (1787341066) [ 2312.350283] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 15:37:53 (1787341073) [ 2318.994735] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 15:37:59 (1787341079) [ 2326.206855] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 15:38:06 (1787341086) [ 2332.647658] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 15:38:13 (1787341093) [ 2339.509164] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 15:38:19 (1787341099) [ 2348.000947] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 15:38:28 (1787341108) [ 2356.262598] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 15:38:36 (1787341116) [ 2365.084254] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 15:38:44 (1787341124) [ 2374.576149] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 15:38:54 (1787341134) [ 2383.566691] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 15:39:03 (1787341143) [ 2390.754554] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 15:39:11 (1787341151) [ 2398.478203] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 15:39:18 (1787341158) [ 2407.459140] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 15:39:28 (1787341168) [ 2414.994894] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 15:39:35 (1787341175) [ 2421.794102] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 15:39:42 (1787341182) [ 2422.088695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2422.131432] LNet: 136717:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2422.145957] LNetError: 136717:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2422.171624] LNet: Removed LNI 192.168.203.57@tcp2 [ 2422.221439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2422.425336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2422.474087] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 2424.036429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2424.073100] LNet: 136859:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2424.084398] LNetError: 136859:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2424.100194] LNet: Removed LNI 192.168.203.57@tcp [ 2424.121763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2424.198973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2424.218767] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 2430.605223] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 15:39:51 (1787341191) [ 2437.532287] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 15:39:58 (1787341198) [ 2444.522442] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 15:40:05 (1787341205) [ 2451.569468] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 15:40:12 (1787341212) [ 2459.062955] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 15:40:19 (1787341219) [ 2466.725524] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 15:40:27 (1787341227) [ 2474.441829] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 15:40:35 (1787341235) [ 2483.029583] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 15:40:43 (1787341243) [ 2492.924253] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 15:40:53 (1787341253) [ 2501.278226] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 15:41:02 (1787341262) [ 2508.217772] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 15:41:09 (1787341269) [ 2514.818269] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 15:41:15 (1787341275) [ 2520.446735] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2279 sec ========== 15:41:21 (1787341281) [ 2520.848201] LNet: 144328:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2520.856291] LNetError: 144328:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2520.879949] LNet: Removed LNI 192.168.203.57@tcp [ 2521.549856] Key type .llcrypt unregistered [ 2521.554115] Key type ._llcrypt unregistered [ 2531.942992] Key type ._llcrypt registered [ 2531.947417] Key type .llcrypt registered [ 2532.350707] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2532.366896] alg: No test for adler32 (adler32-zlib) [ 2533.428171] Lustre: Lustre: Build Version: 2.17.57_44_g85e5bce [ 2533.640745] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 2535.335195] Key type lgssc registered [ 2536.550191] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2594.724041] Lustre: Mounted lustre-client - version 2.17.57_44_g85e5bce [ 2600.273848] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2608.506578] Lustre: Unmounted lustre-client [ 2626.588913] Key type lgssc unregistered [ 2626.922073] LNet: 147286:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2626.935714] LNetError: 147286:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2626.969166] LNet: Removed LNI 192.168.203.57@tcp [ 2627.817291] Key type .llcrypt unregistered [ 2627.821306] Key type ._llcrypt unregistered