[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 503106718 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002377] x2apic enabled [ 0.003011] Switched APIC routing to physical x2apic. [ 0.004014] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.007034] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.008019] pid_max: default: 32768 minimum: 301 [ 0.009210] LSM: Security Framework initializing [ 0.010088] Yama: becoming mindful. [ 0.012054] SELinux: Initializing. [ 0.013151] *** VALIDATE selinux *** [ 0.022462] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027442] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028200] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030095] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031183] *** VALIDATE tmpfs *** [ 0.034181] *** VALIDATE proc *** [ 0.035334] *** VALIDATE cgroup *** [ 0.036024] *** VALIDATE cgroup2 *** [ 0.037000] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037197] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038029] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039041] Spectre V2 : User space: Vulnerable [ 0.040014] Speculative Store Bypass: Vulnerable [ 0.043688] debug: unmapping init [mem 0xffffffff93059000-0xffffffff93060fff] [ 0.046216] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047871] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048035] ... version: 2 [ 0.049020] ... bit width: 48 [ 0.050021] ... generic registers: 4 [ 0.051021] ... value mask: 0000ffffffffffff [ 0.052018] ... max period: 00007fffffffffff [ 0.053022] ... fixed-purpose events: 3 [ 0.054020] ... event mask: 000000070000000f [ 0.056309] rcu: Hierarchical SRCU implementation. [ 0.058803] smp: Bringing up secondary CPUs ... [ 0.059752] x86: Booting SMP configuration: [ 0.060047] .... node #0, CPUs: #1 #2 #3 [ 0.064213] smp: Brought up 1 node, 4 CPUs [ 0.066022] smpboot: Max logical packages: 1 [ 0.067021] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.233036] node 0 deferred pages initialised in 163ms [ 0.237015] devtmpfs: initialized [ 0.238254] x86/mm: Memory block size: 128MB [ 0.240593] gcov: version magic: 0x41383552 [ 0.243427] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.247111] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.249349] pinctrl core: initialized pinctrl subsystem [ 0.251193] [ 0.251651] ************************************************************* [ 0.254015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.255010] ** ** [ 0.257014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.260014] ** ** [ 0.262012] ** This means that this kernel is built to expose internal ** [ 0.264012] ** IOMMU data structures, which may compromise security on ** [ 0.266013] ** your system. ** [ 0.269015] ** ** [ 0.271013] ** If you see this message and you are not debugging the ** [ 0.273010] ** kernel, report this immediately to your vendor! ** [ 0.275012] ** ** [ 0.277009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.279011] ************************************************************* [ 0.281634] NET: Registered protocol family 16 [ 0.282421] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.284118] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.287055] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.290165] cpuidle: using governor menu [ 0.291588] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.294455] PCI: Using configuration type 1 for base access [ 0.296130] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.306130] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.309025] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.313054] cryptd: max_cpu_qlen set to 1000 [ 0.316399] ACPI: Added _OSI(Module Device) [ 0.318024] ACPI: Added _OSI(Processor Device) [ 0.320022] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.323032] ACPI: Added _OSI(Processor Aggregator Device) [ 0.327499] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.335648] ACPI: Interpreter enabled [ 0.337083] ACPI: PM: (supports S0 S3 S4 S5) [ 0.339023] ACPI: Using IOAPIC for interrupt routing [ 0.341165] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.344473] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.356443] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.359054] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.361019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.365162] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.371504] acpiphp: Slot [2] registered [ 0.373220] acpiphp: Slot [5] registered [ 0.374206] acpiphp: Slot [6] registered [ 0.376205] acpiphp: Slot [3] registered [ 0.377122] acpiphp: Slot [4] registered [ 0.379176] acpiphp: Slot [7] registered [ 0.380150] acpiphp: Slot [8] registered [ 0.382149] acpiphp: Slot [9] registered [ 0.383106] acpiphp: Slot [10] registered [ 0.384176] acpiphp: Slot [11] registered [ 0.386123] acpiphp: Slot [12] registered [ 0.387108] acpiphp: Slot [13] registered [ 0.389163] acpiphp: Slot [14] registered [ 0.390199] acpiphp: Slot [15] registered [ 0.392133] acpiphp: Slot [16] registered [ 0.393143] acpiphp: Slot [17] registered [ 0.395249] acpiphp: Slot [18] registered [ 0.397135] acpiphp: Slot [19] registered [ 0.398197] acpiphp: Slot [20] registered [ 0.400148] acpiphp: Slot [21] registered [ 0.402146] acpiphp: Slot [22] registered [ 0.403125] acpiphp: Slot [23] registered [ 0.405163] acpiphp: Slot [24] registered [ 0.407140] acpiphp: Slot [25] registered [ 0.409186] acpiphp: Slot [26] registered [ 0.411147] acpiphp: Slot [27] registered [ 0.413143] acpiphp: Slot [28] registered [ 0.415135] acpiphp: Slot [29] registered [ 0.417154] acpiphp: Slot [30] registered [ 0.418201] acpiphp: Slot [31] registered [ 0.420089] PCI host bridge to bus 0000:00 [ 0.422034] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.425038] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.427031] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.431037] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.434114] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.437050] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.439236] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.442263] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.446580] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.457020] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.460811] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.464026] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.466026] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.469037] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.471745] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.475934] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.479303] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.482900] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.487015] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.498021] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.504017] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.508505] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.516016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.521019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.537015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.547435] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.554983] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.559016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.570989] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.579193] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.582482] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.584402] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.587420] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.590590] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.595076] iommu: Default domain type: Passthrough [ 0.596540] SCSI subsystem initialized [ 0.598170] ACPI: bus type USB registered [ 0.600141] usbcore: registered new interface driver usbfs [ 0.602081] usbcore: registered new interface driver hub [ 0.604089] usbcore: registered new device driver usb [ 0.606250] pps_core: LinuxPPS API ver. 1 registered [ 0.608012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.611066] PTP clock support registered [ 0.613145] EDAC MC: Ver: 3.0.0 [ 0.615144] PCI: Using ACPI for IRQ routing [ 0.616741] NetLabel: Initializing [ 0.618011] NetLabel: domain hash size = 128 [ 0.619007] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.621088] NetLabel: unlabeled traffic allowed by default [ 0.623234] vgaarb: loaded [ 0.624442] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.626014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.632811] clocksource: Switched to clocksource kvm-clock [ 0.737692] VFS: Disk quotas dquot_6.6.0 [ 0.739320] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.741445] *** VALIDATE ramfs *** [ 0.742376] *** VALIDATE hugetlbfs *** [ 0.743393] pnp: PnP ACPI init [ 0.746175] pnp: PnP ACPI: found 6 devices [ 0.761117] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.763957] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.766031] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.768360] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.770601] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.772089] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.774287] NET: Registered protocol family 2 [ 0.776580] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.781582] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.784765] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.789608] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.792670] TCP: Hash tables configured (established 65536 bind 65536) [ 0.795699] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.798287] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.801478] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.805705] NET: Registered protocol family 1 [ 0.809361] RPC: Registered named UNIX socket transport module. [ 0.811819] RPC: Registered udp transport module. [ 0.813787] RPC: Registered tcp transport module. [ 0.815699] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.818848] NET: Registered protocol family 44 [ 0.820735] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.822992] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.825265] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.827794] PCI: CLS 0 bytes, default 64 [ 0.829687] Unpacking initramfs... [ 2.242851] debug: unmapping init [mem 0xffff8dd43cc64000-0xffff8dd43ffcffff] [ 2.247056] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.249314] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.251976] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 2.749983] Initialise system trusted keyrings [ 2.751915] Key type blacklist registered [ 2.753929] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.763894] zbud: loaded [ 2.767790] *** VALIDATE nfs *** [ 2.769159] *** VALIDATE nfs4 *** [ 2.771906] pstore: using deflate compression [ 2.778388] Platform Keyring initialized [ 2.879407] NET: Registered protocol family 38 [ 2.881247] Key type asymmetric registered [ 2.882626] Asymmetric key parser 'x509' registered [ 2.884200] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.886764] io scheduler mq-deadline registered [ 2.888591] io scheduler kyber registered [ 2.890231] io scheduler bfq registered [ 2.891828] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.894875] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.897891] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.900729] ACPI: Power Button [PWRF] [ 2.906228] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.912728] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.923236] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.951210] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.979559] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.984273] Non-volatile memory driver v1.3 [ 2.985776] Linux agpgart interface v0.103 [ 3.018062] virtio_blk virtio1: [vda] 146184 512-byte logical blocks (74.8 MB/71.4 MiB) [ 3.022500] vda: detected capacity change from 0 to 74846208 [ 3.037580] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.040566] vdb: detected capacity change from 0 to 1073741824 [ 3.047571] libphy: Fixed MDIO Bus: probed [ 3.052452] usbcore: registered new interface driver usbserial_generic [ 3.055128] usbserial: USB Serial support registered for generic [ 3.057385] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.061816] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.063754] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.065913] mousedev: PS/2 mouse device common for all mice [ 3.068450] rtc_cmos 00:05: RTC can wake from S4 [ 3.071249] rtc_cmos 00:05: registered as rtc0 [ 3.072753] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.075674] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.075779] intel_pstate: CPU model not supported [ 3.081813] hid: raw HID events driver (C) Jiri Kosina [ 3.081977] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.083950] usbcore: registered new interface driver usbhid [ 3.089424] usbhid: USB HID core driver [ 3.091591] drop_monitor: Initializing network drop monitor service [ 3.094109] Initializing XFRM netlink socket [ 3.094239] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.096183] NET: Registered protocol family 10 [ 3.103569] Segment Routing with IPv6 [ 3.105058] NET: Registered protocol family 17 [ 3.106832] mpls_gso: MPLS GSO support [ 3.112766] RAS: Correctable Errors collector initialized. [ 3.115178] AVX version of gcm_enc/dec engaged. [ 3.116930] AES CTR mode by8 optimization enabled [ 3.185736] sched_clock: Marking stable (3185712472, 0)->(4083541450, -897828978) [ 3.189810] registered taskstats version 1 [ 3.192425] Loading compiled-in X.509 certificates [ 3.194924] zswap: loaded using pool lzo/zbud [ 3.218878] Key type big_key registered [ 3.229253] Key type encrypted registered [ 3.231091] ima: No TPM chip found, activating TPM-bypass! [ 3.233504] ima: Allocated hash algorithm: sha1 [ 3.235294] ima: No architecture policies found [ 3.236932] evm: Initialising EVM extended attributes: [ 3.239059] evm: security.selinux [ 3.239827] evm: security.ima [ 3.240739] evm: security.capability [ 3.242196] evm: HMAC attrs: 0x1 [ 3.243721] rtc_cmos 00:05: setting system clock to 2026-08-19 10:16:26 UTC (1787134586) [ 3.249278] debug: unmapping init [mem 0xffffffff94003000-0xffffffff941fffff] [ 3.251731] debug: unmapping init [mem 0xffffffff92d82000-0xffffffff93058fff] [ 3.260116] Write protecting the kernel read-only data: 28672k [ 3.263211] debug: unmapping init [mem 0xffffffff91403000-0xffffffff915fffff] [ 3.265880] debug: unmapping init [mem 0xffffffff91d14000-0xffffffff91dfffff] [ 3.300692] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.309438] systemd[1]: Detected virtualization kvm. [ 3.311909] systemd[1]: Detected architecture x86-64. [ 3.313881] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.342582] systemd[1]: No hostname configured. [ 3.344973] systemd[1]: Set hostname to . [ 3.347966] random: systemd: uninitialized urandom read (16 bytes read) [ 3.350877] systemd[1]: Initializing machine ID from random generator. [ 3.390812] random: ln: uninitialized urandom read (6 bytes read) [ 3.473986] random: systemd: uninitialized urandom read (16 bytes read) [ 3.478533] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.484776] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.488803] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... Starting Setup Virtual Console... Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Timers. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Journal Service... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Initrd Root Device. [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.155628] device-mapper: uevent: version 1.0.3 [ 4.157954] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.997553] virtio_net virtio0 ens2: renamed from eth0 [ 5.019660] scsi host0: ata_piix [ 5.073296] scsi host1: ata_piix [ 5.077820] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.080412] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 7.478160] random: fast init done [ 8.785160] dracut-initqueue[585]: RTNETLINK answers: File exists [ 9.746728] random: crng init done [ 9.748358] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.253477] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.479075] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.758661] SELinux: Disabled at runtime. [ 11.821254] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.830967] systemd[1]: Detected virtualization kvm. [ 11.833017] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.431483] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.435128] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.441335] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.446338] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.449800] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.458366] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.466607] systemd[1]: Mounting POSIX Message Queue File System... Mounting POSIX Message Queue File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Switch Root. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on Pro[ 12.540352] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS cess Core Dump Socket. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Mounting Kernel Debug File System... Mounting Huge Pages File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-getty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. Starting Remount Root and Kernel File Systems... Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 12.979274] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 13.302139] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.379527] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.497454] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.524637] EDAC sbridge: Ver: 1.1.2 [ 15.043095] Key type dns_resolver registered [ 15.701865] NFS: Registering the id_resolver key type [ 15.704145] Key type id_resolver registered [ 15.705911] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Daily Cleanup of Temporary Directories. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started Login Service. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg426-client login: [ 85.937300] libcfs: loading out-of-tree module taints kernel. [ 85.971668] Key type ._llcrypt registered [ 85.984585] Key type .llcrypt registered [ 86.961264] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 86.999692] alg: No test for adler32 (adler32-zlib) [ 88.574282] Lustre: Lustre: Build Version: 2.17.57_11_g9d7c9d8 [ 89.971134] hrtimer: interrupt took 7119690 ns [ 90.046767] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 91.929427] Key type lgssc registered [ 94.263525] Lustre: Echo OBD driver; http://www.lustre.org/ [ 233.978266] Lustre: Mounted lustre-client - version 2.17.57_11_g9d7c9d8 [ 239.422511] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 257.131385] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing check_logdir /tmp/testlogs/ [ 259.553873] Lustre: lustre-OST0000-osc-ffff8dd4a05a6000: disconnect after 23s idle [ 262.237760] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing yml_node [ 266.472716] Lustre: DEBUG MARKER: Client: 2.17.57.11 [ 269.213335] Lustre: DEBUG MARKER: MDS: 2.17.57.11 [ 272.058824] Lustre: DEBUG MARKER: OSS: 2.17.57.11 [ 273.613289] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Wed Aug 19 06:20:55 EDT 2026 [ 290.812085] Lustre: Unmounted lustre-client [ 310.140507] Key type lgssc unregistered [ 310.580875] LNet: 8419:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 310.593170] LNetError: 8419:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 310.631658] LNet: Removed LNI 192.168.204.26@tcp [ 311.876176] Key type .llcrypt unregistered [ 311.877914] Key type ._llcrypt unregistered [ 343.833237] Key type ._llcrypt registered [ 343.838130] Key type .llcrypt registered [ 344.173173] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 344.197566] alg: No test for adler32 (adler32-zlib) [ 345.155640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 345.268834] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 345.320569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 349.604333] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 349.720810] LNet: 9299:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 349.731499] LNetError: 9299:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 349.761235] LNet: Removed LNI 192.168.204.26@tcp [ 350.441464] Key type .llcrypt unregistered [ 350.444925] Key type ._llcrypt unregistered [ 357.271456] Key type ._llcrypt registered [ 357.275778] Key type .llcrypt registered [ 358.707816] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 06:22:20 (1787134940) [ 359.146165] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 359.166390] alg: No test for adler32 (adler32-zlib) [ 360.191287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 360.281694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 360.299237] LNet: 9724:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 360.316342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 360.415476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 360.446417] LNet: 9731:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 367.988542] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 06:22:29 (1787134949) [ 368.273479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 368.290947] LNet: 10308:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 368.304230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 368.399073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 376.064532] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 06:22:37 (1787134957) [ 376.157240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 376.203284] LNet: 10886:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 376.235638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 376.372968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 382.997990] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 06:22:45 (1787134965) [ 383.170529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 383.201610] LNet: 11479:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 383.229727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 383.306927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 390.183709] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 06:22:51 (1787134971) [ 390.330184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 390.353135] LNet: 12058:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 390.382954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 390.470342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 390.962808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 397.992807] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 06:22:59 (1787134979) [ 398.300203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 398.324972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 398.390452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 398.620662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 399.106089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 406.497814] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 06:23:08 (1787134988) [ 406.730154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 406.759292] LNet: 13252:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 406.767539] LNet: 13252:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 406.792634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 406.850271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 414.813331] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 06:23:16 (1787134996) [ 415.037786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 415.105355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 415.205913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 415.252951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 415.959494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 416.066678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 416.506891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 416.584671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 417.269565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 417.350055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 417.934738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 417.998481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 418.684193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 418.746207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 419.372660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 419.401786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 420.040517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 420.091349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 420.605118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 420.643420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 420.935579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 420.976204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 421.336662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 421.385439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 421.896648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 421.939985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 422.411427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 422.461486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 429.548212] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 06:23:31 (1787135011) [ 429.645323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 429.684729] LNet: 14511:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 429.699626] LNet: 14511:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 429.729310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 429.842409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 429.885327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 437.106050] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 06:23:39 (1787135019) [ 437.492993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 437.557408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 437.672906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 437.724310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 445.065123] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 06:23:46 (1787135026) [ 445.214869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 445.297699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 445.408442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 445.457652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 452.800718] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 06:23:54 (1787135034) [ 452.906166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 452.959645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 453.062102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 453.095569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 459.814417] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 06:24:01 (1787135041) [ 459.999932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 460.059474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 460.156396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 460.209374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 467.409527] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 06:24:09 (1787135049) [ 467.664401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 467.697168] LNet: 17449:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 467.710909] LNet: 17449:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 467.757788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 467.850368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 467.908789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 475.215530] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 06:24:16 (1787135056) [ 475.351510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 475.414528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 475.523753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 476.052991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 482.968174] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 06:24:24 (1787135064) [ 483.130924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 483.187483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 483.458865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 483.733520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 491.044806] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 06:24:32 (1787135072) [ 491.234269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 491.268907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 491.382332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 491.621256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 498.626927] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 06:24:40 (1787135080) [ 498.853219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 498.889399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 499.426624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 499.699624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 505.811644] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 06:24:47 (1787135087) [ 506.026275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 506.066872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 506.145148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 506.216734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 513.214535] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 06:24:55 (1787135095) [ 513.346354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 513.407642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 513.561108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 513.659377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 521.217430] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 06:25:03 (1787135103) [ 521.417484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 521.467594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 521.570587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 527.622494] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 06:25:09 (1787135109) [ 527.713202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 527.750924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 527.828973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 533.825679] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 06:25:15 (1787135115) [ 533.984356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 534.018730] LNet: 22885:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 534.034941] LNet: 22885:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 534.054539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 534.124684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 534.663872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 540.872900] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 06:25:22 (1787135122) [ 540.989983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 541.044871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 541.149842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 541.183581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 547.596143] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 06:25:29 (1787135129) [ 547.707903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 547.808077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 547.952746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 547.994340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 554.729306] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 06:25:36 (1787135136) [ 555.048487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 555.092112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 555.179419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 555.260423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 561.745444] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 06:25:43 (1787135143) [ 562.001622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 562.050384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 562.155544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 562.204812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 569.094319] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 06:25:50 (1787135150) [ 569.426490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 569.477046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 569.544491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 569.623935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 569.647028] LNetError: 25832:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 569.706201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 569.736512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 575.597334] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 06:25:57 (1787135157) [ 575.845847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 575.889661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 575.994593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.3KKp3WfSNx [ 576.092653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.3KKp3WfSNx [ 576.134786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.3KKp3WfSNx [ 582.805566] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 06:26:04 (1787135164) [ 582.981030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 583.036052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 583.124820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 583.195972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 583.269312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 590.681465] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 06:26:12 (1787135172) [ 590.880276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 590.936568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 591.064904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 591.093416] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 592.839680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.26@tcp [ 594.404593] Lustre: Lustre: Build Version: 2.17.57_11_g9d7c9d8 [ 596.448254] Key type lgssc registered [ 597.847783] Lustre: Echo OBD driver; http://www.lustre.org/ [ 601.514337] LNetError: 27593:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.204.26@tcp99 has no available nets [ 604.672567] Key type lgssc unregistered [ 605.027172] LNetError: 28703:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 605.048156] LNet: Removed LNI 192.168.204.26@tcp [ 605.772178] Key type .llcrypt unregistered [ 605.775164] Key type ._llcrypt unregistered [ 612.942572] Key type ._llcrypt registered [ 612.945230] Key type .llcrypt registered [ 614.474774] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 06:26:36 (1787135196) [ 614.866233] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 614.877607] alg: No test for adler32 (adler32-zlib) [ 615.872292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 615.919723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 619.503199] LNet: Message forwarding will be enabled in 2s [ 622.624179] LNet: Message forwarding enabled [ 623.272756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 623.308824] LNet: Message forwarding will stop in 2s to 6s [ 627.893871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 628.562838] LNet: Message forwarding disabled [ 630.743763] LNet: Message forwarding will be enabled in 2s [ 634.725340] LNet: Message forwarding enabled [ 637.872179] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 06:26:59 (1787135219) [ 638.067803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 638.100615] LNet: 30110:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 639.610338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 639.734242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 647.085331] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 06:27:08 (1787135228) [ 647.184067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 647.217402] LNet: 30682:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 647.255995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 647.310558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 647.345203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 647.382502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 647.416749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 647.443279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 647.471866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 647.502573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 647.535629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 647.566875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 647.595886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 647.621767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 647.648992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 647.683027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 647.740365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 647.796128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 647.858211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 647.911548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 647.951543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 647.985858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 648.005279] LNetError: 30734:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 654.329346] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 06:27:16 (1787135236) [ 654.430498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 654.475560] LNet: 31296:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 654.512483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 654.601553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 661.743758] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 06:27:23 (1787135243) [ 663.860681] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg426-server [ 666.217639] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 06:27:27 (1787135247) [ 666.374112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 666.405066] LNet: 32233:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 666.441314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 666.550381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 666.581199] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 667.764539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.204.[27-31]@tcp [ 667.810640] LNet: 32370:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 667.889168] LNetError: 32240:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.29@tcp is being used as a gateway but routing feature is not turned on [ 667.915322] LNetError: 32240:0:(lib-lnet.h:1279:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.29@tcp has gone from up to down [ 668.902948] LNetError: 32240:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.28@tcp is being used as a gateway but routing feature is not turned on [ 668.920165] LNetError: 32240:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 669.923241] LNetError: 32240:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.31@tcp is being used as a gateway but routing feature is not turned on [ 669.948614] LNetError: 32240:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 671.982441] LNetError: 32240:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.29@tcp is being used as a gateway but routing feature is not turned on [ 672.012898] LNetError: 32240:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 675.216401] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 06:27:36 (1787135256) [ 676.086755] LNetError: 32240:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.29@tcp is being used as a gateway but routing feature is not turned on [ 676.112418] LNetError: 32240:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 677.280164] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg426-server [ 679.449949] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 06:27:41 (1787135261) [ 679.590208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 679.628390] LNet: 33301:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 679.633375] LNetError: 33301:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 679.653554] LNet: Removed LNI 192.168.204.26@tcp [ 679.710717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 679.792083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 679.821754] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 680.590097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.204.[27-32/2]@tcp [ 680.629995] LNet: 33424:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 680.644022] LNet: 33424:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 680.663791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.204.[27-32/2]@tcp [ 680.684973] LNetError: 33311:0:(lib-lnet.h:1279:lnet_set_route_aliveness()) route to tcp103 through 192.168.204.27@tcp has gone from up to down [ 680.697989] LNetError: 33311:0:(lib-lnet.h:1279:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 687.636972] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 06:27:49 (1787135269) [ 687.837308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 687.914124] LNet: 34016:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 687.922964] LNetError: 34016:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 687.983140] LNet: Removed LNI 192.168.204.26@tcp [ 688.015519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 688.185393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 688.275292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 688.319572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 688.413489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 688.490289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 688.523167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 688.606220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 688.714887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 688.761287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 688.810900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 688.879574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 688.910719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 688.961667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 689.033951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 689.070635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 689.142128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 689.228359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 689.268666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 689.339591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 689.461022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 689.527525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 689.597129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 689.709331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 689.776223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 689.884952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 690.008169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 690.072493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 690.155637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 690.261190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 690.348866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 690.408869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 690.469119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 690.497786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 690.551546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 696.993777] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 06:27:58 (1787135278) [ 698.862708] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg426-server [ 700.465881] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 06:28:02 (1787135282) [ 702.800171] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg426-server [ 704.205804] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 06:28:06 (1787135286) [ 704.459542] LNet: 35418:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 704.468479] LNet: 35418:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 705.259921] Key type .llcrypt unregistered [ 705.265855] Key type ._llcrypt unregistered [ 712.097497] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 712.227299] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 712.238814] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 712.251412] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 714.753992] Key type ._llcrypt registered [ 714.756639] Key type .llcrypt registered [ 714.948991] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 714.964568] alg: No test for adler32 (adler32-zlib) [ 715.919929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 715.978209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 715.995409] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 716.644853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 716.669819] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 718.722725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 718.742377] LNet: Removed LNI 10.1.2.3@tcp [ 718.873398] LNet: 36138:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 718.882578] LNetError: 36138:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 718.901122] LNet: Removed LNI 192.168.204.26@tcp [ 719.545197] Key type .llcrypt unregistered [ 719.549567] Key type ._llcrypt unregistered [ 731.716382] Key type ._llcrypt registered [ 731.718876] Key type .llcrypt registered [ 733.038962] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 06:28:34 (1787135314) [ 733.555401] Key type .llcrypt unregistered [ 733.562434] Key type ._llcrypt unregistered [ 738.788775] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 738.922447] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 738.937600] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 738.942880] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 741.493768] Key type ._llcrypt registered [ 741.497535] Key type .llcrypt registered [ 741.667676] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 741.680549] alg: No test for adler32 (adler32-zlib) [ 742.602741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 742.661703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 742.699772] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 743.565124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 743.586913] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 746.400927] LNet: 37639:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 746.404286] LNetError: 37639:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 746.409134] LNet: Removed LNI 10.1.2.3@tcp [ 746.421693] LNet: Removed LNI 192.168.204.26@tcp [ 747.167457] Key type .llcrypt unregistered [ 747.172666] Key type ._llcrypt unregistered [ 761.350167] Key type ._llcrypt registered [ 761.352596] Key type .llcrypt registered [ 762.736770] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 06:29:04 (1787135344) [ 763.397293] Key type .llcrypt unregistered [ 763.402839] Key type ._llcrypt unregistered [ 770.289644] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 770.413495] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 770.422632] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 770.426478] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 773.308569] Key type ._llcrypt registered [ 773.312791] Key type .llcrypt registered [ 773.644864] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 773.682696] alg: No test for adler32 (adler32-zlib) [ 774.669815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 774.753357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 774.782855] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 777.370913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 777.395650] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 780.371932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 780.398379] LNet: Removed LNI 10.1.2.3@tcp [ 780.413968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 780.460424] LNet: Removed LNI 10.1.2.31@tcp [ 780.476408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 780.510808] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 781.216420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 781.235309] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 782.410109] LNet: 39568:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 782.432041] LNetError: 39568:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 782.443825] LNet: Removed LNI 10.1.2.31@tcp [ 782.457826] LNet: Removed LNI 10.1.2.3@tcp [ 783.464595] Key type .llcrypt unregistered [ 783.471836] Key type ._llcrypt unregistered [ 797.556531] Key type ._llcrypt registered [ 797.559831] Key type .llcrypt registered [ 799.128609] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 06:29:40 (1787135380) [ 799.597328] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 799.617902] alg: No test for adler32 (adler32-zlib) [ 800.576803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 800.688476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 800.742076] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 801.525068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 801.544128] LNet: 40525:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 801.548847] LNetError: 40525:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 801.560659] LNet: Removed LNI 192.168.204.26@tcp [ 801.578077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 801.642765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.204.26@tcp [ 801.660150] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 802.538790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.204.26@tcp [ 802.589185] LNet: Removed LNI 192.168.204.26@tcp [ 802.667721] LNet: 40660:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 803.383527] Key type .llcrypt unregistered [ 803.391788] Key type ._llcrypt unregistered [ 817.140134] Key type ._llcrypt registered [ 817.146448] Key type .llcrypt registered [ 818.868852] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 06:30:00 (1787135400) [ 821.733319] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg426-server [ 823.825702] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 06:30:05 (1787135405) [ 824.723349] Key type .llcrypt unregistered [ 824.728822] Key type ._llcrypt unregistered [ 832.260172] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 832.359710] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 832.364780] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 832.371929] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 835.189614] Key type ._llcrypt registered [ 835.191473] Key type .llcrypt registered [ 835.447656] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 835.476320] alg: No test for adler32 (adler32-zlib) [ 836.431596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 836.507674] LNet: Added LNI 192.168.204.26@tcp1 [8/256/0/180] [ 837.413172] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 839.950736] LNet: 42508:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 839.959679] LNetError: 42508:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 839.973617] LNet: Removed LNI 192.168.204.26@tcp1 [ 839.988830] LNet: Removed LNI 10.1.2.3@tcp2 [ 840.610419] Key type .llcrypt unregistered [ 840.614897] Key type ._llcrypt unregistered [ 853.102518] Key type ._llcrypt registered [ 853.105673] Key type .llcrypt registered [ 854.664970] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 06:30:36 (1787135436) [ 855.063868] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 855.080881] alg: No test for adler32 (adler32-zlib) [ 856.050654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 856.126878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 856.156781] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 856.987049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 857.037560] LNet: 43481:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 857.045633] LNetError: 43481:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 857.059508] LNet: Removed LNI 192.168.204.26@tcp [ 857.098407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 857.149822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 857.173871] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 865.416384] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 06:30:46 (1787135446) [ 865.525560] LNet: 44063:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 865.529542] LNetError: 44063:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 865.549558] LNet: Removed LNI 192.168.204.26@tcp [ 866.178537] Key type .llcrypt unregistered [ 866.183572] Key type ._llcrypt unregistered [ 872.954865] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 873.129734] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 873.136384] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 873.147661] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 875.980115] Key type ._llcrypt registered [ 875.985281] Key type .llcrypt registered [ 876.302125] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 876.319464] alg: No test for adler32 (adler32-zlib) [ 877.434077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 877.537418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 877.576199] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 878.492193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 878.538597] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 881.538229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 881.620856] LNet: 44819:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 881.638290] LNetError: 44819:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 881.656226] LNet: Removed LNI 10.1.2.3@tcp [ 881.690457] LNet: Removed LNI 192.168.204.26@tcp [ 881.738316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 881.911033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 881.958073] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 881.965443] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 882.527569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 882.549622] LNet: 44846:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 882.559373] LNetError: 44846:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 882.581226] LNetError: 44846:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 882.590551] LNet: Removed LNI 10.1.2.3@tcp [ 882.615788] LNet: Removed LNI 192.168.204.26@tcp [ 882.626917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 882.694980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 882.738988] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 882.745204] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 883.374624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 883.394960] LNet: Added LNI 192.168.204.26@tcp2 [8/256/0/180] [ 884.597191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 884.616670] LNet: 45009:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 884.624995] LNetError: 45009:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 884.632125] LNetError: 45009:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 884.641545] LNet: Removed LNI 10.1.2.3@tcp [ 884.649845] LNet: Removed LNI 192.168.204.26@tcp [ 884.678380] LNet: Removed LNI 192.168.204.26@tcp2 [ 884.702446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 884.827989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 884.872521] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 884.885422] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 884.901119] LNet: Added LNI 192.168.204.26@tcp2 [8/256/0/180] [ 893.393171] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 06:31:14 (1787135474) [ 893.754430] LNet: 45606:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 893.758753] LNetError: 45606:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 893.762614] LNetError: 45606:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 893.770934] LNet: Removed LNI 10.1.2.3@tcp [ 893.773709] LNet: Removed LNI 192.168.204.26@tcp [ 893.780847] LNet: Removed LNI 192.168.204.26@tcp2 [ 894.572922] Key type .llcrypt unregistered [ 894.575295] Key type ._llcrypt unregistered [ 904.261969] Key type ._llcrypt registered [ 904.268274] Key type .llcrypt registered [ 904.652073] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 904.684229] alg: No test for adler32 (adler32-zlib) [ 905.677435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 905.776649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 905.806408] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 906.932872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 906.997315] LNet: 46005:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 907.004510] LNetError: 46005:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 907.033854] LNet: Removed LNI 192.168.204.26@tcp [ 907.110554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 907.220411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 907.256286] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 908.018422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 908.036321] LNet: 46035:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 908.043977] LNetError: 46035:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 908.072490] LNet: Removed LNI 192.168.204.26@tcp [ 908.097638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 908.161666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 908.177663] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 908.664882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 908.731261] LNet: Removed LNI 192.168.204.26@tcp [ 908.776766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 908.825623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 908.861239] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 909.241315] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 909.420651] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 909.432335] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 909.448199] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 912.158117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 912.176457] LNet: 46147:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 912.185200] LNet: 46147:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 912.201188] LNetError: 46147:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 912.216785] LNetError: 46147:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 912.261987] LNet: Removed LNI 192.168.204.26@tcp [ 912.307949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 912.426662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 912.451691] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 919.627794] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 06:31:41 (1787135501) [ 919.680136] LNet: 46736:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 919.699136] LNetError: 46736:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 919.717208] LNet: Removed LNI 192.168.204.26@tcp [ 920.205559] Key type .llcrypt unregistered [ 920.209451] Key type ._llcrypt unregistered [ 927.113356] Key type ._llcrypt registered [ 927.128124] Key type .llcrypt registered [ 927.511766] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 927.530473] alg: No test for adler32 (adler32-zlib) [ 928.578782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 928.638266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 928.677480] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 929.538487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 929.556518] LNet: 47146:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 929.561949] LNetError: 47146:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 929.575671] LNet: Removed LNI 192.168.204.26@tcp [ 929.602408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 929.688626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 929.735195] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 936.770813] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 06:31:58 (1787135518) [ 936.997598] LNet: 47738:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 937.011561] LNetError: 47738:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 937.027924] LNet: Removed LNI 192.168.204.26@tcp [ 937.766888] Key type .llcrypt unregistered [ 937.772866] Key type ._llcrypt unregistered [ 943.707959] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 943.781248] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 943.784753] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 943.793517] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 946.456988] Key type ._llcrypt registered [ 946.459544] Key type .llcrypt registered [ 946.646502] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 946.658740] alg: No test for adler32 (adler32-zlib) [ 947.617593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 947.669986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 947.690837] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 948.725774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 948.766959] LNet: 48191:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 948.780796] LNetError: 48191:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 948.801435] LNet: Removed LNI 192.168.204.26@tcp [ 948.819397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 948.921664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 949.000270] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 957.663776] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 06:32:19 (1787135539) [ 958.009365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 958.048046] LNet: 48791:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 958.071068] LNetError: 48791:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 958.120148] LNet: Removed LNI 192.168.204.26@tcp [ 958.187783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 958.380969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 966.994503] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 06:32:28 (1787135548) [ 967.102724] LNet: 49369:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 967.588812] Key type .llcrypt unregistered [ 967.591722] Key type ._llcrypt unregistered [ 973.332579] Key type ._llcrypt registered [ 973.335371] Key type .llcrypt registered [ 973.557892] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 973.593979] alg: No test for adler32 (adler32-zlib) [ 974.507419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 974.566504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 974.612240] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 975.430936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 975.462937] LNet: 49775:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 975.470492] LNetError: 49775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 975.517710] LNet: Removed LNI 192.168.204.26@tcp [ 975.540603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 975.597243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 975.613583] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 982.237191] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 06:32:43 (1787135563) [ 982.509654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 982.529224] LNet: 50368:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 982.533307] LNetError: 50368:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 982.542873] LNet: Removed LNI 192.168.204.26@tcp [ 982.567254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 982.669396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 989.100041] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 06:32:50 (1787135570) [ 990.511378] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 992.574323] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 06:32:54 (1787135574) [ 994.280711] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 996.402182] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 06:32:57 (1787135577) [ 998.228969] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 999.978789] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 06:33:01 (1787135581) [ 1001.627238] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1003.263208] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 06:33:05 (1787135585) [ 1004.726777] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1006.134050] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 06:33:08 (1787135588) [ 1006.282523] LNet: 52418:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1006.934832] Key type .llcrypt unregistered [ 1006.939847] Key type ._llcrypt unregistered [ 1013.686390] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1013.899182] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1013.913111] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1013.928513] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1016.366405] Key type ._llcrypt registered [ 1016.368223] Key type .llcrypt registered [ 1016.556429] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1016.573331] alg: No test for adler32 (adler32-zlib) [ 1017.626642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1017.728177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1017.748554] LNet: Added LNI 192.168.204.26@tcp [8/512/0/180] [ 1017.754438] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1017.757751] LNet: 52746:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1017.830692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1017.850824] LNet: 52765:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1017.857041] LNetError: 52765:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1017.864566] LNet: Removed LNI 10.1.2.3@tcp [ 1017.866539] LNetError: 52765:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1017.877684] LNet: Removed LNI 192.168.204.26@tcp [ 1017.899458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1017.995941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1018.066919] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 1018.168700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1018.201205] LNet: 52789:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1018.217065] LNetError: 52789:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1018.254120] LNet: Removed LNI 192.168.204.26@tcp [ 1018.271697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1018.380745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1018.424200] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 1018.429106] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1018.439335] LNet: 52797:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1018.616473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1018.675190] LNet: 52815:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1018.689940] LNetError: 52815:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1018.698250] LNet: Removed LNI 10.1.2.3@tcp [ 1018.702806] LNetError: 52815:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1018.750672] LNet: Removed LNI 192.168.204.26@tcp [ 1018.789104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1018.901349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1018.928411] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 1018.944501] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1018.946442] LNet: 52824:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1019.132859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1019.154081] LNet: 52850:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1019.160802] LNetError: 52850:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1019.167462] LNet: Removed LNI 10.1.2.3@tcp [ 1019.170780] LNetError: 52850:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1019.179917] LNet: Removed LNI 192.168.204.26@tcp [ 1019.199313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1019.299259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1019.330921] LNet: Added LNI 192.168.204.26@tcp [8/256/0/182] [ 1019.338471] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1019.520229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1019.543296] LNet: 52878:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1019.554801] LNetError: 52878:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1019.560542] LNet: Removed LNI 10.1.2.3@tcp [ 1019.564841] LNetError: 52878:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1019.584885] LNet: Removed LNI 192.168.204.26@tcp [ 1019.636799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1019.735505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1019.775168] LNet: Added LNI 192.168.204.26@tcp [8/256/0/182] [ 1019.785898] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1019.792495] LNet: 52886:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1019.921834] LNet: Removed LNI 10.1.2.3@tcp [ 1019.955259] LNet: Removed LNI 192.168.204.26@tcp [ 1020.902957] Key type .llcrypt unregistered [ 1020.904278] Key type ._llcrypt unregistered [ 1033.421450] Key type ._llcrypt registered [ 1033.423812] Key type .llcrypt registered [ 1035.183110] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 06:33:36 (1787135616) [ 1036.822502] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 1038.528295] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 06:33:40 (1787135620) [ 1038.985062] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1039.012236] alg: No test for adler32 (adler32-zlib) [ 1039.987481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1040.090848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1040.118098] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 1040.935433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.204.26 [ 1040.972412] LNet: 54155:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1040.994826] LNet: Removed LNI 192.168.204.26@tcp [ 1047.273462] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 06:33:49 (1787135629) [ 1047.418479] LNet: 54717:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1047.442608] LNetError: 54717:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1047.468962] LNet: Removed LNI 192.168.204.26@tcp [ 1048.420702] Key type .llcrypt unregistered [ 1048.426294] Key type ._llcrypt unregistered [ 1055.524218] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1055.856802] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1058.538279] Key type ._llcrypt registered [ 1058.540824] Key type .llcrypt registered [ 1058.749391] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1058.763156] alg: No test for adler32 (adler32-zlib) [ 1060.108675] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1067.927239] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 06:34:09 (1787135649) [ 1068.122625] LNet: 55635:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1068.141775] LNetError: 55635:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1068.172231] LNet: Removed LNI 10.1.2.3@tcp [ 1069.007474] Key type .llcrypt unregistered [ 1069.013506] Key type ._llcrypt unregistered [ 1076.194855] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1076.424813] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1076.429032] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1076.433543] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1079.123637] Key type ._llcrypt registered [ 1079.126181] Key type .llcrypt registered [ 1079.279681] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1079.294148] alg: No test for adler32 (adler32-zlib) [ 1080.179081] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1086.167204] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 06:34:27 (1787135667) [ 1087.671162] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1089.306321] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 06:34:31 (1787135671) [ 1089.429178] LNet: 56825:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1089.435446] LNetError: 56825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1089.478467] LNet: Removed LNI 10.1.2.3@tcp [ 1090.010070] Key type .llcrypt unregistered [ 1090.014453] Key type ._llcrypt unregistered [ 1096.071729] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1096.299172] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1096.311058] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1096.313233] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1098.979552] Key type ._llcrypt registered [ 1098.982591] Key type .llcrypt registered [ 1099.229119] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1099.245202] alg: No test for adler32 (adler32-zlib) [ 1100.141210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1100.212060] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1108.304888] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 06:34:50 (1787135690) [ 1108.733146] LNet: 58063:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1108.746068] LNetError: 58063:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1108.775905] LNet: Removed LNI 10.1.2.3@tcp [ 1109.483606] Key type .llcrypt unregistered [ 1109.487644] Key type ._llcrypt unregistered [ 1115.457425] Key type ._llcrypt registered [ 1115.460332] Key type .llcrypt registered [ 1115.775381] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1115.801934] alg: No test for adler32 (adler32-zlib) [ 1116.675877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1116.695931] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 1125.223724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.126@tcp [ 1126.404447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp -d 192.168.204.126@tcp -m GET -e local_interrupt -r 1 [ 1126.488433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.26@tcp -m GET -e local_interrupt -r 1 [ 1126.546535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.126@tcp [ 1126.588663] LNet: There was an unexpected network error while writing to 192.168.204.126: rc = -22 [ 1126.948676] LNet: 1 local NIs in recovery (showing 1): 192.168.204.26@tcp [ 1127.013698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1127.056566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1127.094385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1127.134419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1127.172405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1127.215347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1127.246360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1127.297837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1127.360244] LNet: 58505:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1127.421976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1127.453661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1127.486535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1127.522097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1127.556354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1127.586523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1127.633693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1128.253239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp -d 192.168.204.126@tcp -m GET -e local_dropped -r 1 [ 1128.307602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp -d 192.168.204.26@tcp -m GET -e local_dropped -r 1 [ 1128.385422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.126@tcp [ 1128.442445] LNet: There was an unexpected network error while writing to 192.168.204.126: rc = -22 [ 1128.774586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1128.806389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1128.874649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1128.945143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1128.999562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1129.094788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1129.136257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1129.190778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1129.214511] LNet: 58615:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1129.236580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1129.272730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1129.313311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1129.353287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1129.388969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1129.474730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1129.524587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1130.306257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp -d 192.168.204.126@tcp -m GET -e local_aborted -r 1 [ 1130.372848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.26@tcp -m GET -e local_aborted -r 1 [ 1130.471480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.126@tcp [ 1130.520050] LNet: There was an unexpected network error while writing to 192.168.204.126: rc = -22 [ 1130.859341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1130.923270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1130.991849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1131.061362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1131.137288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1131.187110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1131.238518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1131.281731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1131.320348] LNet: 58724:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1131.358714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1131.384298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1131.409623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1131.434822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1131.465474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1131.498183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1131.532046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1132.267673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp -d 192.168.204.126@tcp -m GET -e local_no_route -r 1 [ 1132.299299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.26@tcp -m GET -e local_no_route -r 1 [ 1132.348546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.126@tcp [ 1132.668207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1132.721078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1132.756401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1132.795388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1132.838452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1132.886393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1132.925741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1132.980814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1133.053896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1133.129679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1133.179816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1133.264278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1133.347370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1133.436267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1133.481134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1134.206218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp -d 192.168.204.126@tcp -m GET -e local_timeout -r 1 [ 1134.277132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.26@tcp -m GET -e local_timeout -r 1 [ 1134.327254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.126@tcp [ 1134.377819] LNet: There was an unexpected network error while writing to 192.168.204.126: rc = -22 [ 1134.390032] LNet: Skipped 2 previous similar messages [ 1134.784583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1134.821904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1134.849617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1134.880636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1134.905746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1134.935592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1134.959118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1134.981331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1135.006164] LNet: 58942:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1135.018078] LNet: 58942:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1135.055900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1135.126694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1135.184215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1135.225195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1135.259992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1135.296352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1135.338388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1135.904571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.126@tcp -m GET -e local_error -r 1 [ 1135.934520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp -d 192.168.204.26@tcp -m GET -e local_error -r 1 [ 1135.970164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.126@tcp [ 1136.284321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1136.320905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1136.365253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1136.399539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1136.454541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1136.497202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1136.526327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1136.560211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1136.590984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1136.615487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1136.654324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1136.706250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1136.772672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1136.796523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1136.840331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1137.141168] LNet: 59097:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1137.147743] LNetError: 59097:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1137.172255] LNet: Removed LNI 192.168.204.26@tcp [ 1137.558892] Key type .llcrypt unregistered [ 1137.567020] Key type ._llcrypt unregistered [ 1155.115759] Key type ._llcrypt registered [ 1155.121564] Key type .llcrypt registered [ 1156.702381] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 06:35:38 (1787135738) [ 1157.571274] Key type .llcrypt unregistered [ 1157.573165] Key type ._llcrypt unregistered [ 1164.512521] Key type ._llcrypt registered [ 1164.515688] Key type .llcrypt registered [ 1164.755739] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1164.771392] alg: No test for adler32 (adler32-zlib) [ 1165.730995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1165.784637] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 1173.835474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.126@tcp [ 1184.951686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1185.009546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1185.026654] LNet: Added LNI 192.168.204.26@tcp1 [8/256/0/180] [ 1186.576815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp -d 192.168.204.126@tcp -m GET -e local_interrupt -r 1 [ 1186.662785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.126@tcp1 -m GET -e local_interrupt -r 1 [ 1186.710883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp -d 192.168.204.26@tcp -m GET -e local_interrupt -r 1 [ 1186.765800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp -d 192.168.204.26@tcp1 -m GET -e local_interrupt -r 1 [ 1186.808413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp1 -d 192.168.204.126@tcp -m GET -e local_interrupt -r 1 [ 1186.853301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp1 -d 192.168.204.126@tcp1 -m GET -e local_interrupt -r 1 [ 1186.894280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp1 -d 192.168.204.26@tcp -m GET -e local_interrupt -r 1 [ 1186.945871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp1 -d 192.168.204.26@tcp1 -m GET -e local_interrupt -r 1 [ 1187.001311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.126@tcp [ 1187.027638] LNet: There was an unexpected network error while writing to 192.168.204.126: rc = -22 [ 1187.038265] LNet: 1 local NIs in recovery (showing 1): 192.168.204.26@tcp [ 1187.278429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1187.311587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1187.334372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1187.359987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1187.382032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1187.417459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1187.447785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1187.487510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1187.519405] LNet: 60568:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1187.539909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1187.582674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1187.614241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1187.640991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1187.679049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1187.712201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1187.748686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1188.367444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.126@tcp -m GET -e local_dropped -r 1 [ 1188.421699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp -d 192.168.204.126@tcp1 -m GET -e local_dropped -r 1 [ 1188.489493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.26@tcp -m GET -e local_dropped -r 1 [ 1188.563697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp -d 192.168.204.26@tcp1 -m GET -e local_dropped -r 1 [ 1188.609860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp1 -d 192.168.204.126@tcp -m GET -e local_dropped -r 1 [ 1188.649742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp1 -d 192.168.204.126@tcp1 -m GET -e local_dropped -r 1 [ 1188.690536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp1 -d 192.168.204.26@tcp -m GET -e local_dropped -r 1 [ 1188.734425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp1 -d 192.168.204.26@tcp1 -m GET -e local_dropped -r 1 [ 1188.793403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.126@tcp [ 1188.831395] LNet: There was an unexpected network error while writing to 192.168.204.126: rc = -22 [ 1188.839904] LNet: Skipped 2 previous similar messages [ 1189.147859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1189.208580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1189.236908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1189.271263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1189.299559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1189.329328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1189.366387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1189.393102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1189.413745] LNet: 60690:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1189.433404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1189.481312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1189.520440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1189.548534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1189.598364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1189.630240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1189.657538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1190.314840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.126@tcp -m GET -e local_aborted -r 1 [ 1190.404676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp -d 192.168.204.126@tcp1 -m GET -e local_aborted -r 1 [ 1190.447890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.26@tcp -m GET -e local_aborted -r 1 [ 1190.501831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.26@tcp1 -m GET -e local_aborted -r 1 [ 1190.554573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp1 -d 192.168.204.126@tcp -m GET -e local_aborted -r 1 [ 1190.619981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp1 -d 192.168.204.126@tcp1 -m GET -e local_aborted -r 1 [ 1190.685222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp1 -d 192.168.204.26@tcp -m GET -e local_aborted -r 1 [ 1190.730618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp1 -d 192.168.204.26@tcp1 -m GET -e local_aborted -r 1 [ 1190.793402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.126@tcp [ 1190.820888] LNet: There was an unexpected network error while writing to 192.168.204.126: rc = -22 [ 1190.827056] LNet: Skipped 2 previous similar messages [ 1191.107979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1191.138447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1191.171172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1191.227944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1191.277844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1191.310326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1191.341367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1191.396725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1191.422833] LNet: 60811:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1191.447530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1191.487915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1191.551807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1191.593248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1191.646515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1191.699629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1191.744679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1192.416308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp -d 192.168.204.126@tcp -m GET -e local_no_route -r 1 [ 1192.501707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp -d 192.168.204.126@tcp1 -m GET -e local_no_route -r 1 [ 1192.549551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp -d 192.168.204.26@tcp -m GET -e local_no_route -r 1 [ 1192.629397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.26@tcp1 -m GET -e local_no_route -r 1 [ 1192.676481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp1 -d 192.168.204.126@tcp -m GET -e local_no_route -r 1 [ 1192.733093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp1 -d 192.168.204.126@tcp1 -m GET -e local_no_route -r 1 [ 1192.769188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp1 -d 192.168.204.26@tcp -m GET -e local_no_route -r 1 [ 1192.812859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp1 -d 192.168.204.26@tcp1 -m GET -e local_no_route -r 1 [ 1192.873860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.126@tcp [ 1192.903493] LNet: There was an unexpected network error while writing to 192.168.204.126: rc = -22 [ 1192.910528] LNet: Skipped 2 previous similar messages [ 1193.212780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1193.258407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1193.296336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1193.331693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1193.364519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1193.461471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1193.490683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1193.547464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1193.575619] LNet: 60932:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1193.595667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1193.636854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1193.688753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1193.715421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1193.752305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1193.793284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1193.841997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1194.502128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp -d 192.168.204.126@tcp -m GET -e local_timeout -r 1 [ 1194.548071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp -d 192.168.204.126@tcp1 -m GET -e local_timeout -r 1 [ 1194.583510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.26@tcp -m GET -e local_timeout -r 1 [ 1194.657584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp -d 192.168.204.26@tcp1 -m GET -e local_timeout -r 1 [ 1194.711922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp1 -d 192.168.204.126@tcp -m GET -e local_timeout -r 1 [ 1194.750242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp1 -d 192.168.204.126@tcp1 -m GET -e local_timeout -r 1 [ 1194.790813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp1 -d 192.168.204.26@tcp -m GET -e local_timeout -r 1 [ 1194.819826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp1 -d 192.168.204.26@tcp1 -m GET -e local_timeout -r 1 [ 1194.854825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.126@tcp [ 1195.096652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1195.123145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1195.157199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1195.186481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1195.218106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1195.264077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1195.296346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1195.342191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1195.381868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1195.431828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1195.474664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1195.516211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1195.558607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1195.588310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1195.626414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1196.063989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.126@tcp -m GET -e local_error -r 1 [ 1196.094761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.126@tcp1 -m GET -e local_error -r 1 [ 1196.125740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp -d 192.168.204.26@tcp -m GET -e local_error -r 1 [ 1196.154721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp -d 192.168.204.26@tcp1 -m GET -e local_error -r 1 [ 1196.186869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp1 -d 192.168.204.126@tcp -m GET -e local_error -r 1 [ 1196.263921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp1 -d 192.168.204.126@tcp1 -m GET -e local_error -r 1 [ 1196.317752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp1 -d 192.168.204.26@tcp -m GET -e local_error -r 1 [ 1196.369237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp1 -d 192.168.204.26@tcp1 -m GET -e local_error -r 1 [ 1196.415344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.126@tcp [ 1196.681360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1196.707358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1196.732913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1196.752593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1196.778584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1196.804880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1196.842984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1196.885131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1196.935347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1196.964555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1196.994846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1197.031259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1197.061714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1197.094219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1197.122757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1202.345566] LNet: 61255:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1202.351477] LNetError: 61255:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1202.357539] LNet: Removed LNI 192.168.204.26@tcp [ 1202.368809] LNet: Removed LNI 192.168.204.26@tcp1 [ 1202.694777] Key type .llcrypt unregistered [ 1202.699863] Key type ._llcrypt unregistered [ 1218.266301] Key type ._llcrypt registered [ 1218.267812] Key type .llcrypt registered [ 1219.417961] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 06:36:41 (1787135801) [ 1220.283707] Key type .llcrypt unregistered [ 1220.289297] Key type ._llcrypt unregistered [ 1226.217431] Key type ._llcrypt registered [ 1226.226695] Key type .llcrypt registered [ 1226.442570] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1226.459190] alg: No test for adler32 (adler32-zlib) [ 1227.373609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1227.412379] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 1235.123842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.126@tcp [ 1235.908875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.126@tcp -m GET -e remote_dropped -r 1 [ 1235.946300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.26@tcp -m GET -e remote_dropped -r 1 [ 1235.975601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.126@tcp [ 1235.993374] LNet: There was an unexpected network error while writing to 192.168.204.126: rc = -22 [ 1236.176957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1236.216314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1236.254837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1236.282037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1236.321574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1236.358930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1236.403727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1236.457962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1236.476816] LNet: 62507:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1236.500593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1236.552123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1236.605096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1236.652498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1236.693528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1236.745917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1236.789448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1237.487242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.126@tcp -m GET -e remote_error -r 1 [ 1237.527305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.26@tcp -m GET -e remote_error -r 1 [ 1237.588427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.126@tcp [ 1237.644454] LNet: There was an unexpected network error while writing to 192.168.204.126: rc = -22 [ 1237.961837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1237.994495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1238.029325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1238.087506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1238.134696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1238.183955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1238.239143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1238.284586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1238.328724] LNet: 62616:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1238.368631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1238.413863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1238.460603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1238.509401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1238.542916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1238.583987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1238.620773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1239.286439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.126@tcp -m GET -e remote_timeout -r 1 [ 1239.353349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp -d 192.168.204.26@tcp -m GET -e remote_timeout -r 1 [ 1239.404319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.126@tcp [ 1239.446482] LNet: There was an unexpected network error while writing to 192.168.204.126: rc = -22 [ 1239.678989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1239.712528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1239.738693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1239.766910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1239.799764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1239.840151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1239.878524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1239.921498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1239.937211] LNet: 62725:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1239.954563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1240.001757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1240.065875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1240.141834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1240.181640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1240.216772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1240.298350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1240.686480] LNet: 62770:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1240.692723] LNetError: 62770:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1240.715672] LNet: Removed LNI 192.168.204.26@tcp [ 1241.105659] Key type .llcrypt unregistered [ 1241.109093] Key type ._llcrypt unregistered [ 1257.213201] Key type ._llcrypt registered [ 1257.216640] Key type .llcrypt registered [ 1258.443927] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 06:37:20 (1787135840) [ 1259.515589] Key type .llcrypt unregistered [ 1259.520456] Key type ._llcrypt unregistered [ 1265.904332] Key type ._llcrypt registered [ 1265.910462] Key type .llcrypt registered [ 1266.067767] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1266.089090] alg: No test for adler32 (adler32-zlib) [ 1267.012635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1267.064581] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 1274.649164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.126@tcp [ 1284.205543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1284.241783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1284.265443] LNet: Added LNI 192.168.204.26@tcp1 [8/256/0/180] [ 1285.760798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp -d 192.168.204.126@tcp -m GET -e remote_dropped -r 1 [ 1285.820355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.126@tcp1 -m GET -e remote_dropped -r 1 [ 1285.884803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.26@tcp -m GET -e remote_dropped -r 1 [ 1285.930498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp -d 192.168.204.26@tcp1 -m GET -e remote_dropped -r 1 [ 1285.971803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp1 -d 192.168.204.126@tcp -m GET -e remote_dropped -r 1 [ 1286.021778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp1 -d 192.168.204.126@tcp1 -m GET -e remote_dropped -r 1 [ 1286.075876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp1 -d 192.168.204.26@tcp -m GET -e remote_dropped -r 1 [ 1286.143488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp1 -d 192.168.204.26@tcp1 -m GET -e remote_dropped -r 1 [ 1286.179298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.126@tcp [ 1286.202635] LNet: There was an unexpected network error while writing to 192.168.204.126: rc = -22 [ 1286.207857] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.126@tcp [ 1286.461405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1286.503108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1286.532651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1286.561992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1286.592353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1286.618631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1286.640834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1286.664913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1286.680290] LNet: 64240:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1286.693242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1286.721476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1286.764204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1286.791879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1286.841506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1286.864834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1286.903828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1286.946477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1287.390202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp -d 192.168.204.126@tcp -m GET -e remote_error -r 1 [ 1287.438086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.126@tcp1 -m GET -e remote_error -r 1 [ 1287.474523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.26@tcp -m GET -e remote_error -r 1 [ 1287.504949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.26@tcp1 -m GET -e remote_error -r 1 [ 1287.535755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp1 -d 192.168.204.126@tcp -m GET -e remote_error -r 1 [ 1287.580991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp1 -d 192.168.204.126@tcp1 -m GET -e remote_error -r 1 [ 1287.640561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp1 -d 192.168.204.26@tcp -m GET -e remote_error -r 1 [ 1287.697174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp1 -d 192.168.204.26@tcp1 -m GET -e remote_error -r 1 [ 1287.766480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.126@tcp [ 1287.796645] LNet: There was an unexpected network error while writing to 192.168.204.126: rc = -22 [ 1287.803528] LNet: Skipped 2 previous similar messages [ 1288.037227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1288.058754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1288.090097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1288.136743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1288.199486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1288.254951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1288.306383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1288.358275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1288.382032] LNet: 64363:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1288.413713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1288.468643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1288.498901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1288.535689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1288.558257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1288.583763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1288.610225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1288.629753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1289.104454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp -d 192.168.204.126@tcp -m GET -e remote_timeout -r 1 [ 1289.131643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp -d 192.168.204.126@tcp1 -m GET -e remote_timeout -r 1 [ 1289.159155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.26@tcp -m GET -e remote_timeout -r 1 [ 1289.191901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp -d 192.168.204.26@tcp1 -m GET -e remote_timeout -r 1 [ 1289.224407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp1 -d 192.168.204.126@tcp -m GET -e remote_timeout -r 1 [ 1289.260976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp1 -d 192.168.204.126@tcp1 -m GET -e remote_timeout -r 1 [ 1289.303484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp1 -d 192.168.204.26@tcp -m GET -e remote_timeout -r 1 [ 1289.348871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp1 -d 192.168.204.26@tcp1 -m GET -e remote_timeout -r 1 [ 1289.409995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.126@tcp [ 1289.446403] LNet: There was an unexpected network error while writing to 192.168.204.126: rc = -22 [ 1289.455514] LNet: Skipped 1 previous similar message [ 1289.701655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1289.725801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1289.750249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1289.772316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1289.794781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1289.815511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1289.836943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1289.876577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1289.903414] LNet: 64486:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1289.924323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1289.965972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1290.040944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1290.085710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1290.137914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1290.189272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1290.226043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1290.282062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1295.380921] LNet: 64568:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1295.398797] LNetError: 64568:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1295.412250] LNet: Removed LNI 192.168.204.26@tcp [ 1295.434126] LNet: Removed LNI 192.168.204.26@tcp1 [ 1295.900530] Key type .llcrypt unregistered [ 1295.904796] Key type ._llcrypt unregistered [ 1312.724440] Key type ._llcrypt registered [ 1312.730457] Key type .llcrypt registered [ 1314.101955] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 06:38:15 (1787135895) [ 1315.808521] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1317.505572] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 06:38:19 (1787135899) [ 1318.400525] Key type .llcrypt unregistered [ 1318.402118] Key type ._llcrypt unregistered [ 1323.797835] Key type ._llcrypt registered [ 1323.799669] Key type .llcrypt registered [ 1323.963737] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1323.981654] alg: No test for adler32 (adler32-zlib) [ 1324.900796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1324.926840] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 1332.805210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.126@tcp [ 1333.456221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.126@tcp -m GET -e network_timeout -r 1 [ 1333.541488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.26@tcp -m GET -e network_timeout -r 1 [ 1333.584452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.126@tcp [ 1333.609061] LNet: There was an unexpected network error while writing to 192.168.204.126: rc = -22 [ 1333.882227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1333.912152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1333.946714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1334.017583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1334.103897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1334.135919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1334.166646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1334.180408] LNet: 1 local NIs in recovery (showing 1): 192.168.204.26@tcp [ 1334.183146] LNet: There was an unexpected network error while writing to 192.168.204.26: rc = -22 [ 1334.216826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1334.240631] LNet: 66117:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1334.279493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1334.313308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1334.341621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1334.375939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1334.415394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1334.471906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1334.754266] LNet: 66160:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1334.761985] LNetError: 66160:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1334.788078] LNet: Removed LNI 192.168.204.26@tcp [ 1335.268441] Key type .llcrypt unregistered [ 1335.271762] Key type ._llcrypt unregistered [ 1352.845048] Key type ._llcrypt registered [ 1352.849647] Key type .llcrypt registered [ 1353.106332] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1353.141406] alg: No test for adler32 (adler32-zlib) [ 1354.116379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1354.164452] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 1362.611423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.126@tcp [ 1370.219445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1370.274951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1370.305400] LNet: Added LNI 192.168.204.26@tcp1 [8/256/0/180] [ 1371.763353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp -d 192.168.204.126@tcp -m GET -e network_timeout -r 1 [ 1371.825526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp -d 192.168.204.126@tcp1 -m GET -e network_timeout -r 1 [ 1371.885415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp -d 192.168.204.26@tcp -m GET -e network_timeout -r 1 [ 1371.937587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.26@tcp1 -m GET -e network_timeout -r 1 [ 1372.003593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp1 -d 192.168.204.126@tcp -m GET -e network_timeout -r 1 [ 1372.057711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp1 -d 192.168.204.126@tcp1 -m GET -e network_timeout -r 1 [ 1372.123888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp1 -d 192.168.204.26@tcp -m GET -e network_timeout -r 1 [ 1372.196746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp1 -d 192.168.204.26@tcp1 -m GET -e network_timeout -r 1 [ 1372.252807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.126@tcp [ 1372.298656] LNet: There was an unexpected network error while writing to 192.168.204.126: rc = -22 [ 1372.567807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1372.598391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1372.627690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1372.642413] LNet: 1 local NIs in recovery (showing 1): 192.168.204.26@tcp [ 1372.654229] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.126@tcp [ 1372.669026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1372.692433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1372.717679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1372.747727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1372.792534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1372.819778] LNet: 67043:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1372.863637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1372.894722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1372.923745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1372.965263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1373.001193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1373.039787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1377.731980] LNet: There was an unexpected network error while writing to 192.168.204.126: rc = -22 [ 1378.132630] LNet: 67120:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1378.145409] LNetError: 67120:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1378.165288] LNet: Removed LNI 192.168.204.26@tcp [ 1378.180095] LNet: Removed LNI 192.168.204.26@tcp1 [ 1378.557525] Key type .llcrypt unregistered [ 1378.559965] Key type ._llcrypt unregistered [ 1396.830341] Key type ._llcrypt registered [ 1396.840390] Key type .llcrypt registered [ 1398.318897] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 06:39:39 (1787135979) [ 1398.668175] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1398.685288] alg: No test for adler32 (adler32-zlib) [ 1399.659751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1399.726666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1399.775500] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 1400.778815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1400.809119] LNet: Added LNI 192.168.204.26@tcp1 [8/256/0/180] [ 1401.744434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.26@tcp [ 1401.819183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1401.857672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1401.902826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1401.947938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1401.987277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1402.012995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.26@tcp [ 1402.031030] LNetError: 68222:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.204.26@tcp to recovery [ 1402.784324] LNet: 1 local NIs in recovery (showing 1): 192.168.204.26@tcp [ 1402.785328] LNet: There was an unexpected network error while writing to 192.168.204.26: rc = -22 [ 1404.832850] LNet: There was an unexpected network error while writing to 192.168.204.26: rc = -22 [ 1405.571135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1408.933036] LNet: There was an unexpected network error while writing to 192.168.204.26: rc = -22 [ 1409.114937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1409.153482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1409.208995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1409.225558] LNet: 68329:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1409.229783] LNetError: 68329:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1410.272174] LNet: Removed LNI 192.168.204.26@tcp [ 1410.273911] LNetError: 68329:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1410.289715] LNet: Removed LNI 192.168.204.26@tcp1 [ 1410.318904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1410.392406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1410.427934] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 1411.067391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1411.082731] LNet: Added LNI 192.168.204.26@tcp1 [8/256/0/180] [ 1411.894268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.26@tcp [ 1411.964649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1412.038162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1412.115382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1412.152611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1412.189474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1412.231722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1412.265601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.26@tcp [ 1412.299489] LNetError: 68602:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.204.26@tcp to recovery [ 1412.385335] LNet: There was an unexpected network error while writing to 192.168.204.26: rc = -22 [ 1414.774603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1418.531119] LNet: There was an unexpected network error while writing to 192.168.204.26: rc = -22 [ 1418.546908] LNet: Skipped 1 previous similar message [ 1422.916902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1422.949494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1422.996676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1430.466678] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 06:40:12 (1787136012) [ 1430.591801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1430.649813] LNet: 69303:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1430.668479] LNetError: 69303:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1430.681234] LNet: Removed LNI 192.168.204.26@tcp [ 1430.705847] LNet: Removed LNI 192.168.204.26@tcp1 [ 1430.782502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1430.859984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1430.880117] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 1431.980207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1432.030603] LNet: Added LNI 192.168.204.26@tcp1 [8/256/0/180] [ 1433.298649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.26@tcp [ 1433.365745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1433.408980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1433.447597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1433.504980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1433.570569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1433.631499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.26@tcp --health 0 [ 1433.804590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1433.889770] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.26@tcp [ 1434.917644] LNet: There was an unexpected network error while writing to 192.168.204.26: rc = -22 [ 1434.931667] LNet: Skipped 1 previous similar message [ 1448.568877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1448.761898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1448.812468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1448.884673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.26@tcp --health 500 [ 1448.986754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1454.597733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1454.698734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1454.755274] LNet: 69785:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1454.762647] LNetError: 69785:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1454.775031] LNetError: 69785:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1454.784897] LNet: Removed LNI 192.168.204.26@tcp [ 1454.809556] LNet: Removed LNI 192.168.204.26@tcp1 [ 1454.850693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1454.908132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1454.931697] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 1455.643733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1455.688695] LNet: Added LNI 192.168.204.26@tcp1 [8/256/0/180] [ 1456.601668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.26@tcp [ 1456.655092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1456.717213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1456.753462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1456.796678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1456.848859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1456.905415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1456.949619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.26@tcp --health 0 [ 1457.952719] LNet: There was an unexpected network error while writing to 192.168.204.26: rc = -22 [ 1457.967670] LNet: Skipped 3 previous similar messages [ 1458.293597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1468.519900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1468.540243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1468.566908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1476.215468] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 06:40:57 (1787136057) [ 1476.766141] LNet: 70830:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1476.789969] LNetError: 70830:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1476.796560] LNetError: 70830:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1476.800587] LNet: Removed LNI 192.168.204.26@tcp [ 1476.807919] LNet: Removed LNI 192.168.204.26@tcp1 [ 1477.649701] Key type .llcrypt unregistered [ 1477.661748] Key type ._llcrypt unregistered [ 1486.806439] Key type ._llcrypt registered [ 1486.809606] Key type .llcrypt registered [ 1487.030561] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1487.058808] alg: No test for adler32 (adler32-zlib) [ 1487.986763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1488.024218] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 1499.288942] Lustre: DEBUG MARKER: Initial discovery [ 1499.328879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.126@tcp [ 1501.812627] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1501.869738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1501.944922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.126@tcp [ 1501.983543] LNet: There was an unexpected network error while writing to 192.168.204.126: rc = -22 [ 1502.017845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1502.080427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.126@tcp -r 1 -m GET -l 3 [ 1502.141837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.204.126@tcp -r 1 -m GET -e local_error [ 1502.196983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.126@tcp -r 1 -m PUT -l 6 [ 1502.369709] LNet: 1 local NIs in recovery (showing 1): 192.168.204.26@tcp [ 1503.974785] Lustre: DEBUG MARKER: Force oleg426-client.virtnet to discover 192.168.204.126@tcp (in background) [ 1504.074162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.126@tcp [ 1505.696909] Lustre: DEBUG MARKER: Force oleg426-server to discover 192.168.204.26@tcp [ 1507.168246] LNet: There was an unexpected network error while writing to 192.168.204.126: rc = -22 [ 1508.426907] Lustre: DEBUG MARKER: Wait for 71589 [ 1510.326927] Lustre: DEBUG MARKER: Finished wait on 71589 [ 1510.494377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1510.519736] LNet: There was an unexpected network error while writing to 192.168.204.126: rc = -22 [ 1510.545221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1510.980857] LNet: 72052:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1510.987739] LNetError: 72052:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1511.013441] LNet: Removed LNI 192.168.204.26@tcp [ 1511.504757] Key type .llcrypt unregistered [ 1511.511578] Key type ._llcrypt unregistered [ 1529.284112] Key type ._llcrypt registered [ 1529.287747] Key type .llcrypt registered [ 1530.691496] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 06:41:52 (1787136112) [ 1531.451544] Key type .llcrypt unregistered [ 1531.453418] Key type ._llcrypt unregistered [ 1537.962617] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1538.091182] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1538.098275] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1538.109251] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1540.785377] Key type ._llcrypt registered [ 1540.794637] Key type .llcrypt registered [ 1540.995885] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1541.015666] alg: No test for adler32 (adler32-zlib) [ 1542.027391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1542.113398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1542.148222] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 1543.009689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1543.045615] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1552.837572] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 06:42:14 (1787136134) [ 1553.002796] LNet: 74136:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1553.008923] LNetError: 74136:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1553.016697] LNet: Removed LNI 10.1.2.3@tcp [ 1553.032324] LNet: Removed LNI 192.168.204.26@tcp [ 1553.669450] Key type .llcrypt unregistered [ 1553.675244] Key type ._llcrypt unregistered [ 1561.753849] Key type ._llcrypt registered [ 1561.755127] Key type .llcrypt registered [ 1561.912344] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1561.928981] alg: No test for adler32 (adler32-zlib) [ 1562.793943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1562.827408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1562.845143] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 1563.465678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1563.491380] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1564.688168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.204.26@tcp [ 1565.729228] LNetError: Unexpected error -22 connecting to 192.168.204.26@tcp at host 192.168.204.26:988 [ 1571.415877] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 06:42:33 (1787136153) [ 1571.709695] LNet: 75271:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1571.729250] LNetError: 75271:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1571.738865] LNet: Removed LNI 10.1.2.3@tcp [ 1571.748057] LNet: Removed LNI 192.168.204.26@tcp [ 1572.515798] Key type .llcrypt unregistered [ 1572.522529] Key type ._llcrypt unregistered [ 1579.441036] Key type ._llcrypt registered [ 1579.443275] Key type .llcrypt registered [ 1579.689340] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1579.730770] alg: No test for adler32 (adler32-zlib) [ 1580.592097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1580.640914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1580.663900] LNet: Added LNI 192.168.204.26@tcp1 [8/256/0/180] [ 1581.284868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1581.302245] LNet: Added LNI 192.168.204.26@tcp2 [8/256/0/180] [ 1582.099152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.204.26@tcp1 --nid 192.168.204.26@tcp2 [ 1586.149860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.26@tcp1 192.168.204.26@tcp2 [ 1586.167478] LNetError: 76111:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.204.26@tcp1 to 192.168.204.26@tcp2 - Route table may be misconfigured [ 1586.190329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.26@tcp2 192.168.204.26@tcp1 [ 1593.195822] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 06:42:54 (1787136174) [ 1593.381541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1593.410272] LNet: 76681:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1593.421950] LNetError: 76681:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1594.464237] LNet: Removed LNI 192.168.204.26@tcp1 [ 1594.469469] LNetError: 76681:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1594.491065] LNet: Removed LNI 192.168.204.26@tcp2 [ 1594.520428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1594.615882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1594.657862] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 1595.524810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1595.558229] LNet: Added LNI 192.168.204.26@tcp1 [8/256/0/180] [ 1596.432355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.26@tcp [ 1596.463965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.26@tcp 192.168.204.26@tcp [ 1596.507864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.26@tcp1 192.168.204.26@tcp1 [ 1596.551959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.26@tcp -d 192.168.204.26@tcp -e network_timeout [ 1596.626840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.26@tcp -d 192.168.204.26@tcp1 -e network_timeout [ 1596.683960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.26@tcp1 -d 192.168.204.26@tcp -e network_timeout [ 1596.736246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.26@tcp1 -d 192.168.204.26@tcp1 -e network_timeout [ 1596.805438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.26@tcp [ 1596.822927] LNet: There was an unexpected network error while writing to 192.168.204.26: rc = -22 [ 1596.840304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1596.933211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1597.009904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1597.664555] LNet: 1 local NIs in recovery (showing 1): 192.168.204.26@tcp [ 1602.852581] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 06:43:04 (1787136184) [ 1602.963284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1602.992638] LNet: 77538:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1603.004290] LNetError: 77538:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1603.019813] LNet: Removed LNI 192.168.204.26@tcp [ 1604.071130] LNet: Removed LNI 192.168.204.26@tcp1 [ 1604.110794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1604.206561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1604.461312] LNet: 77586:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1604.790213] Key type .llcrypt unregistered [ 1604.792110] Key type ._llcrypt unregistered [ 1615.975360] Key type ._llcrypt registered [ 1615.977202] Key type .llcrypt registered [ 1617.039651] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 06:43:19 (1787136199) [ 1617.509373] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1617.526853] alg: No test for adler32 (adler32-zlib) [ 1618.488737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1619.728370] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1621.184527] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 06:43:23 (1787136203) [ 1621.324712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1621.359217] LNet: 78684:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1621.397661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1621.476675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1621.509761] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 1622.305556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1622.327594] LNet: Added LNI 192.168.204.26@tcp1 [8/256/0/180] [ 1623.103345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.26@tcp [ 1623.137274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.26@tcp1 [ 1623.184336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.26@tcp1 [ 1628.856859] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 06:43:30 (1787136210) [ 1630.460618] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg426-server [ 1632.677844] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 06:43:34 (1787136214) [ 1634.778415] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg426-server [ 1636.644360] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 06:43:38 (1787136218) [ 1638.070162] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg426-server [ 1639.639043] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 06:43:41 (1787136221) [ 1641.484481] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg426-server [ 1643.261562] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 06:43:45 (1787136225) [ 1645.169972] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg426-server [ 1646.823418] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 06:43:48 (1787136228) [ 1648.461270] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg426-server [ 1650.119613] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 06:43:52 (1787136232) [ 1651.762742] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg426-server [ 1653.378497] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 06:43:55 (1787136235) [ 1654.965790] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg426-server [ 1656.684245] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 06:43:58 (1787136238) [ 1658.417371] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg426-server [ 1660.267810] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 06:44:01 (1787136241) [ 1660.537829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1660.614334] LNet: 82561:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1660.636440] LNetError: 82561:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1660.659274] LNet: Removed LNI 192.168.204.26@tcp [ 1661.747402] LNet: Removed LNI 192.168.204.26@tcp1 [ 1661.784514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1661.908396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1661.955258] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 1662.823360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1662.917615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.26@tcp [ 1663.066555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1663.086487] LNet: 82708:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1663.093357] LNetError: 82708:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1663.103924] LNetError: 82708:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1663.126995] LNet: Removed LNI 192.168.204.26@tcp [ 1663.149763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1663.194808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1663.215878] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 1663.976092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.26@tcp [ 1665.369834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1665.430155] LNet: 82862:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1665.454703] LNetError: 82862:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1666.472734] LNet: Removed LNI 192.168.204.26@tcp [ 1666.503813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1666.556608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1666.616614] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 1667.305437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1667.381341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.26@tcp [ 1667.586327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1667.622708] LNetError: 83010:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1667.640897] LNet: Removed LNI 192.168.204.26@tcp [ 1667.664864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1667.727811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1667.750946] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 1668.554558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.26@tcp [ 1668.783461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1669.869507] LNet: Removed LNI 192.168.204.26@tcp [ 1669.897431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1669.993321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1670.054433] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 1670.896489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1670.963712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.26@tcp [ 1672.318893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1672.336448] LNet: 83311:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1672.340919] LNet: 83311:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1672.346492] LNetError: 83311:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1672.353887] LNetError: 83311:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1673.382947] LNet: Removed LNI 192.168.204.26@tcp [ 1673.432993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1673.527196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1673.563435] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 1674.451388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.26@tcp [ 1674.727748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1675.814252] LNet: Removed LNI 192.168.204.26@tcp [ 1675.888936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1675.968567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1676.000844] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 1676.864422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1676.969441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.26@tcp [ 1677.266566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1678.383914] LNet: Removed LNI 192.168.204.26@tcp [ 1678.421983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1678.517709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1678.555736] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 1679.474551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.26@tcp [ 1679.762430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1680.807699] LNet: Removed LNI 192.168.204.26@tcp [ 1680.858221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1680.924344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1680.956475] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 1681.847362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1681.961630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.26@tcp [ 1683.420633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1683.445541] LNet: 83906:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1683.455364] LNet: 83906:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1683.477524] LNetError: 83906:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1683.489077] LNetError: 83906:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1684.518780] LNet: Removed LNI 192.168.204.26@tcp [ 1684.577542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1684.639927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1684.685949] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 1685.518940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.26@tcp [ 1686.987325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1688.043069] LNet: Removed LNI 192.168.204.26@tcp [ 1688.095641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1688.159565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1688.202724] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 1688.972151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1689.050411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.26@tcp [ 1689.383865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1690.478903] LNet: Removed LNI 192.168.204.26@tcp [ 1690.533226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1690.614353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1690.649721] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 1691.546846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.26@tcp [ 1691.952806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1693.039387] LNet: Removed LNI 192.168.204.26@tcp [ 1693.104401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1693.177526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1693.228577] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 1694.195459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1694.250262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1694.284439] LNet: Removed LNI 192.168.204.26@tcp [ 1694.308427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1694.386534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1694.412147] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 1695.500783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1695.599293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1695.629827] LNet: Removed LNI 192.168.204.26@tcp [ 1695.663557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1695.722897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1695.821907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1695.862475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1695.936358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1696.040216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1696.076725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1696.134576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1696.234704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1696.296632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1696.419914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1696.456584] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 1697.692305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1697.762408] LNet: Removed LNI 192.168.204.26@tcp [ 1697.786097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1697.945957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1705.368184] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 06:44:47 (1787136287) [ 1705.573508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1705.593194] LNet: 85375:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1705.597742] LNet: 85375:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 1705.618899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1705.674752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1705.696353] LNet: Added LNI 192.168.204.26@tcp231 [8/256/0/180] [ 1706.996109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1707.016139] LNetError: 85514:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1707.022814] LNetError: 85514:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1707.037519] LNet: Removed LNI 192.168.204.26@tcp231 [ 1707.074884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1707.163844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1707.214525] LNet: Added LNI 192.168.204.26@tcp231 [8/256/0/180] [ 1707.229127] LNet: 85522:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1707.948721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1707.981510] LNet: Removed LNI 192.168.204.26@tcp231 [ 1708.005682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1708.050027] LNet: Added LNI 192.168.204.26@tcp231 [8/256/0/180] [ 1709.669417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1709.715250] LNet: Removed LNI 192.168.204.26@tcp231 [ 1709.746783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1709.838782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1709.869632] LNet: Added LNI 192.168.204.26@tcp231 [8/256/0/0] [ 1709.873955] LNet: 85674:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1710.342737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1710.387880] LNet: Removed LNI 192.168.204.26@tcp231 [ 1710.415850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1710.479848] LNet: Added LNI 192.168.204.26@tcp231 [8/256/0/0] [ 1712.013563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1712.048551] LNet: Removed LNI 192.168.204.26@tcp231 [ 1712.082112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1712.158983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1712.206898] LNet: Added LNI 192.168.204.26@tcp231 [8/256/0/60] [ 1712.219789] LNet: 85826:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1712.708760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1712.741706] LNet: Removed LNI 192.168.204.26@tcp231 [ 1712.768112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1712.816104] LNet: Added LNI 192.168.204.26@tcp231 [8/256/0/60] [ 1714.377794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1714.433713] LNet: Removed LNI 192.168.204.26@tcp231 [ 1714.476040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1714.558811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1714.612119] LNet: Added LNI 192.168.204.26@tcp231 [8/256/0/180] [ 1714.619152] LNet: 85977:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1715.122302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1715.189893] LNet: Removed LNI 192.168.204.26@tcp231 [ 1715.223463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1715.266932] LNet: Added LNI 192.168.204.26@tcp231 [8/256/0/180] [ 1723.553562] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 06:45:05 (1787136305) [ 1723.676269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1723.730048] LNet: Removed LNI 192.168.204.26@tcp231 [ 1723.778643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1723.870441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1723.929389] LNet: Added LNI 192.168.204.26@tcp232 [8/256/0/180] [ 1724.755978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1724.797962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1724.842177] LNet: Removed LNI 192.168.204.26@tcp232 [ 1724.857995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1724.887056] LNet: Added LNI 192.168.204.26@tcp232 [8/256/0/180] [ 1725.603870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1725.645592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1725.692309] LNet: Removed LNI 192.168.204.26@tcp232 [ 1725.709452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1725.731530] LNet: Added LNI 192.168.204.26@tcp232 [8/256/0/180] [ 1726.581976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1726.621779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1726.656476] LNet: Removed LNI 192.168.204.26@tcp232 [ 1726.684449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1726.708039] LNet: Added LNI 192.168.204.26@tcp232 [8/256/0/180] [ 1727.509911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1727.546460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1727.573440] LNet: Removed LNI 192.168.204.26@tcp232 [ 1727.591336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1733.594522] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 06:45:15 (1787136315) [ 1734.647923] Key type .llcrypt unregistered [ 1734.650723] Key type ._llcrypt unregistered [ 1741.434364] Key type ._llcrypt registered [ 1741.438156] Key type .llcrypt registered [ 1741.670882] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1741.716599] alg: No test for adler32 (adler32-zlib) [ 1742.689713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1742.711470] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 1751.193679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.126@tcp [ 1762.499484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1762.583524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1762.645719] LNet: Added LNI 192.168.204.26@tcp1 [8/256/0/180] [ 1764.282213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp -d 192.168.204.126@tcp -m GET -e local_interrupt -r 2 [ 1764.340863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp -d 192.168.204.126@tcp1 -m GET -e local_interrupt -r 2 [ 1764.388993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.26@tcp -m GET -e local_interrupt -r 2 [ 1764.453844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.26@tcp1 -m GET -e local_interrupt -r 2 [ 1764.508215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp1 -d 192.168.204.126@tcp -m GET -e local_interrupt -r 2 [ 1764.550446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp1 -d 192.168.204.126@tcp1 -m GET -e local_interrupt -r 2 [ 1764.613968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp1 -d 192.168.204.26@tcp -m GET -e local_interrupt -r 2 [ 1764.699303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp1 -d 192.168.204.26@tcp1 -m GET -e local_interrupt -r 2 [ 1764.757346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.26@tcp 192.168.204.126@tcp [ 1764.794887] LNet: There was an unexpected network error while writing to 192.168.204.126: rc = -22 [ 1764.799683] LNet: 1 local NIs in recovery (showing 1): 192.168.204.26@tcp [ 1765.157160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1765.202517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1765.256967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1765.319908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1765.356878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1765.412361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1765.450653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1765.488601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1765.521772] LNet: 88482:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1765.550656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1765.600694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1765.646676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1765.687405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1765.740342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1765.795900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1766.411969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp -d 192.168.204.126@tcp -m GET -e local_dropped -r 2 [ 1766.440892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.126@tcp1 -m GET -e local_dropped -r 2 [ 1766.471854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp -d 192.168.204.26@tcp -m GET -e local_dropped -r 2 [ 1766.505685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.26@tcp1 -m GET -e local_dropped -r 2 [ 1766.540151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp1 -d 192.168.204.126@tcp -m GET -e local_dropped -r 2 [ 1766.573652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp1 -d 192.168.204.126@tcp1 -m GET -e local_dropped -r 2 [ 1766.607933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp1 -d 192.168.204.26@tcp -m GET -e local_dropped -r 2 [ 1766.639645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp1 -d 192.168.204.26@tcp1 -m GET -e local_dropped -r 2 [ 1766.671945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.26@tcp 192.168.204.126@tcp [ 1766.689857] LNet: There was an unexpected network error while writing to 192.168.204.126: rc = -22 [ 1766.977724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1767.017271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1767.049637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1767.079406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1767.108839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1767.149622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1767.188967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1767.218980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1767.244870] LNet: 88602:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1767.261788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1767.319641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1767.375546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1767.401709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1767.430370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1767.459623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1768.087534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp -d 192.168.204.126@tcp -m GET -e local_aborted -r 2 [ 1768.124570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp -d 192.168.204.126@tcp1 -m GET -e local_aborted -r 2 [ 1768.163486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp -d 192.168.204.26@tcp -m GET -e local_aborted -r 2 [ 1768.211750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.26@tcp1 -m GET -e local_aborted -r 2 [ 1768.246869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp1 -d 192.168.204.126@tcp -m GET -e local_aborted -r 2 [ 1768.295502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp1 -d 192.168.204.126@tcp1 -m GET -e local_aborted -r 2 [ 1768.337794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp1 -d 192.168.204.26@tcp -m GET -e local_aborted -r 2 [ 1768.388369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp1 -d 192.168.204.26@tcp1 -m GET -e local_aborted -r 2 [ 1768.428505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.26@tcp 192.168.204.126@tcp [ 1768.454959] LNet: There was an unexpected network error while writing to 192.168.204.126: rc = -22 [ 1768.767837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1768.801614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1768.841455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1768.868374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1768.896831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1768.926947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1768.951929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1768.980797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1768.995209] LNet: 88722:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1769.010773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1769.047601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1769.090424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1769.124421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1769.150363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1769.179719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1769.631623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.126@tcp -m GET -e local_no_route -r 2 [ 1769.670880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp -d 192.168.204.126@tcp1 -m GET -e local_no_route -r 2 [ 1769.719453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.26@tcp -m GET -e local_no_route -r 2 [ 1769.775483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp -d 192.168.204.26@tcp1 -m GET -e local_no_route -r 2 [ 1769.833994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp1 -d 192.168.204.126@tcp -m GET -e local_no_route -r 2 [ 1769.873568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp1 -d 192.168.204.126@tcp1 -m GET -e local_no_route -r 2 [ 1769.913472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp1 -d 192.168.204.26@tcp -m GET -e local_no_route -r 2 [ 1769.942600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp1 -d 192.168.204.26@tcp1 -m GET -e local_no_route -r 2 [ 1769.971645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.26@tcp 192.168.204.126@tcp [ 1770.271551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1770.302847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1770.331222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1770.360425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1770.399993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1770.441130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1770.490769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1770.529173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1770.562833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1770.599965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1770.660835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1770.700646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1770.738961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1770.782132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1771.442180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.126@tcp -m GET -e local_timeout -r 2 [ 1771.486429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.126@tcp1 -m GET -e local_timeout -r 2 [ 1771.536865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.26@tcp -m GET -e local_timeout -r 2 [ 1771.592733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp -d 192.168.204.26@tcp1 -m GET -e local_timeout -r 2 [ 1771.637123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp1 -d 192.168.204.126@tcp -m GET -e local_timeout -r 2 [ 1771.681944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp1 -d 192.168.204.126@tcp1 -m GET -e local_timeout -r 2 [ 1771.717904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp1 -d 192.168.204.26@tcp -m GET -e local_timeout -r 2 [ 1771.764181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp1 -d 192.168.204.26@tcp1 -m GET -e local_timeout -r 2 [ 1771.844385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.26@tcp 192.168.204.126@tcp [ 1771.890548] LNet: There was an unexpected network error while writing to 192.168.204.126: rc = -22 [ 1771.914087] LNet: Skipped 1 previous similar message [ 1772.170652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1772.190265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1772.213805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1772.248235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1772.298116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1772.330552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1772.358096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1772.394868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1772.416940] LNet: 88963:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1772.427674] LNet: 88963:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1772.449354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1772.498788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1772.546549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1772.586394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1772.638184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1772.693337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1773.304465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.126@tcp -m GET -e remote_dropped -r 2 [ 1773.338524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.126@tcp1 -m GET -e remote_dropped -r 2 [ 1773.372262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp -d 192.168.204.26@tcp -m GET -e remote_dropped -r 2 [ 1773.403418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp -d 192.168.204.26@tcp1 -m GET -e remote_dropped -r 2 [ 1773.462238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp1 -d 192.168.204.126@tcp -m GET -e remote_dropped -r 2 [ 1773.517896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp1 -d 192.168.204.126@tcp1 -m GET -e remote_dropped -r 2 [ 1773.587372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.26@tcp1 -d 192.168.204.26@tcp -m GET -e remote_dropped -r 2 [ 1773.619792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.26@tcp1 -d 192.168.204.26@tcp1 -m GET -e remote_dropped -r 2 [ 1773.671796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.26@tcp 192.168.204.126@tcp [ 1773.704710] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.126@tcp [ 1774.083343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1774.123331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1774.158798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1774.194784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1774.228475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1774.264498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1774.295113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1774.334902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1774.366408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1774.397141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1774.426559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1774.467655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1774.528697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1774.571956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1781.100586] LNet: 89161:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1781.109531] LNetError: 89161:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1781.120442] LNet: Removed LNI 192.168.204.26@tcp [ 1781.140885] LNet: Removed LNI 192.168.204.26@tcp1 [ 1781.603723] Key type .llcrypt unregistered [ 1781.610618] Key type ._llcrypt unregistered [ 1799.160589] Key type ._llcrypt registered [ 1799.162778] Key type .llcrypt registered [ 1800.482621] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 06:46:22 (1787136382) [ 1800.941751] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1800.953483] alg: No test for adler32 (adler32-zlib) [ 1801.924770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1801.991055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1802.020293] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 1802.921806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1802.956705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.26@tcp [ 1802.983426] LNetError: 90140:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.204.26@tcp to recovery [ 1803.071440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.204.26@tcp [ 1803.175196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1809.745378] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 06:46:31 (1787136391) [ 1810.027493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1810.049821] LNet: 90732:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1810.056972] LNetError: 90732:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1810.073232] LNet: Removed LNI 192.168.204.26@tcp [ 1810.102351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1810.187963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1810.241826] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 1811.063650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1811.087237] LNet: Added LNI 192.168.204.26@tcp1 [8/256/0/180] [ 1812.292814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.204.26@tcp [ 1812.314380] LNetError: 90998:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.204.26@tcp to recovery [ 1813.216368] LNet: 1 local NIs in recovery (showing 1): 192.168.204.26@tcp [ 1813.220539] LNet: There was an unexpected network error while writing to 192.168.204.26: rc = -22 [ 1813.575135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.204.26@tcp [ 1822.185124] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 06:46:44 (1787136404) [ 1824.009353] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg426-server [ 1825.666652] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 06:46:47 (1787136407) [ 1827.919909] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg426-server [ 1829.577953] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 06:46:51 (1787136411) [ 1831.534501] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg426-server [ 1833.398403] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 06:46:55 (1787136415) [ 1833.521955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1833.540861] LNet: 92624:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1833.549987] LNetError: 92624:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1833.555522] LNet: Removed LNI 192.168.204.26@tcp [ 1833.563290] LNet: Removed LNI 192.168.204.26@tcp1 [ 1833.612838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1833.683305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1833.707106] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 1834.325593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1834.379413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1834.420808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1834.492388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1834.567977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1834.622229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1840.385071] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 06:47:02 (1787136422) [ 1840.498912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1840.532616] LNet: 93350:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1840.541846] LNetError: 93350:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1840.558142] LNetError: 93350:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1840.606049] LNet: Removed LNI 192.168.204.26@tcp [ 1840.621720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1840.703320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1840.738808] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 1846.823868] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 06:47:08 (1787136428) [ 1846.884660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1846.903447] LNet: 94051:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1846.912401] LNetError: 94051:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1846.934550] LNet: Removed LNI 192.168.204.26@tcp [ 1846.976783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1847.054192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1847.084389] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 1853.778471] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 06:47:15 (1787136435) [ 1855.107561] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1856.857724] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 06:47:18 (1787136438) [ 1857.390965] LNet: 95093:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1857.400267] LNetError: 95093:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1857.413874] LNet: Removed LNI 192.168.204.26@tcp [ 1858.066115] Key type .llcrypt unregistered [ 1858.069974] Key type ._llcrypt unregistered [ 1864.320137] Key type ._llcrypt registered [ 1864.323985] Key type .llcrypt registered [ 1864.563842] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1864.594299] alg: No test for adler32 (adler32-zlib) [ 1865.518345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1865.582341] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 1873.887632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.126@tcp [ 1879.606959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.204.126@tcp [ 1880.071992] LNet: 95519:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1880.084273] LNetError: 95519:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1880.104948] LNet: Removed LNI 192.168.204.26@tcp [ 1880.643489] Key type .llcrypt unregistered [ 1880.649590] Key type ._llcrypt unregistered [ 1892.454790] Key type ._llcrypt registered [ 1892.466272] Key type .llcrypt registered [ 1894.189312] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1895.787385] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1897.511807] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 06:47:59 (1787136479) [ 1899.327958] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1901.038138] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 06:48:02 (1787136482) [ 1903.335870] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg426-server [ 1905.466815] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 06:48:07 (1787136487) [ 1907.613171] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg426-server [ 1909.389656] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 06:48:11 (1787136491) [ 1910.944063] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1910.964991] alg: No test for adler32 (adler32-zlib) [ 1912.117876] Lustre: Lustre: Build Version: 2.17.57_11_g9d7c9d8 [ 1912.419415] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 1914.184951] Key type lgssc registered [ 1915.644419] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1937.861929] Key type lgssc unregistered [ 1938.107294] LNet: 98699:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1938.111829] LNetError: 98699:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1938.139208] LNet: Removed LNI 192.168.204.26@tcp [ 1938.849581] Key type .llcrypt unregistered [ 1938.852896] Key type ._llcrypt unregistered [ 1954.587410] Key type ._llcrypt registered [ 1954.589939] Key type .llcrypt registered [ 1956.028473] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 06:48:57 (1787136537) [ 1956.538889] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1956.573371] alg: No test for adler32 (adler32-zlib) [ 1957.508865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1957.577483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1957.619371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1957.667283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1957.715993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1964.198193] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 06:49:06 (1787136546) [ 1964.466516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1964.493431] LNet: 100115:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1964.515096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1964.570919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1964.603672] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 1965.543580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1965.570451] LNet: 100259:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1965.580393] LNetError: 100259:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1965.598656] LNet: Removed LNI 192.168.204.26@tcp [ 1965.627056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1965.709750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1965.738242] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 1966.440345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1966.470669] LNetError: 100287:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1966.490027] LNet: Removed LNI 192.168.204.26@tcp [ 1966.516176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1966.578766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1966.597900] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 1973.538964] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 06:49:15 (1787136555) [ 1973.633777] LNet: 100872:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1973.638304] LNet: 100872:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1973.644616] LNetError: 100872:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1973.671905] LNet: Removed LNI 192.168.204.26@tcp [ 1974.179865] Key type .llcrypt unregistered [ 1974.181832] Key type ._llcrypt unregistered [ 1981.478499] Key type ._llcrypt registered [ 1981.481190] Key type .llcrypt registered [ 1981.700333] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1981.732414] alg: No test for adler32 (adler32-zlib) [ 1982.652934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1982.732568] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 1982.735750] LNet: *** cfs_fail_loc=e003, val=100*** [ 1983.758060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.26@tcp [ 1983.815651] LNet: *** cfs_fail_loc=e003, val=100*** [ 1983.817944] LNet: Skipped 2 previous similar messages [ 1983.830757] Ping reply received with mapping: [ 1983.832609] found metadata reply NI[192.168.204.26@tcp] [ 1983.840446] 1a cc a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1983.859051] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1983.871375] 5a 5a 5a 5a 5a 5a 5a 5a [ 1984.070833] LNet: 101295:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1984.076853] LNetError: 101295:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1984.096393] LNet: Removed LNI 192.168.204.26@tcp [ 1985.079409] Key type .llcrypt unregistered [ 1985.082193] Key type ._llcrypt unregistered [ 1999.430901] Key type ._llcrypt registered [ 1999.435242] Key type .llcrypt registered [ 2001.105363] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 06:49:42 (1787136582) [ 2001.761349] Key type .llcrypt unregistered [ 2001.763577] Key type ._llcrypt unregistered [ 2007.714923] Key type ._llcrypt registered [ 2007.717871] Key type .llcrypt registered [ 2007.956920] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2007.977678] alg: No test for adler32 (adler32-zlib) [ 2009.190720] LNetError: 102353:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2009.798346] Key type .llcrypt unregistered [ 2009.800524] Key type ._llcrypt unregistered [ 2014.744838] Key type ._llcrypt registered [ 2014.747735] Key type .llcrypt registered [ 2016.062783] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 06:49:57 (1787136597) [ 2016.633592] Key type .llcrypt unregistered [ 2016.636210] Key type ._llcrypt unregistered [ 2023.732642] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2023.745611] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2026.542363] Key type ._llcrypt registered [ 2026.544586] Key type .llcrypt registered [ 2026.778174] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2026.788292] alg: No test for adler32 (adler32-zlib) [ 2027.678615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2027.741829] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2038.599399] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 06:50:20 (1787136620) [ 2038.961679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2038.983213] LNet: 104143:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2038.987685] LNetError: 104143:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2039.007461] LNet: Removed LNI 10.1.2.3@tcp [ 2039.049386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2040.162468] Key type .llcrypt unregistered [ 2040.164174] Key type ._llcrypt unregistered [ 2049.301265] Key type ._llcrypt registered [ 2049.302749] Key type .llcrypt registered [ 2049.611599] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2049.657936] alg: No test for adler32 (adler32-zlib) [ 2050.596278] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 2050.737251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.26@tcp [ 2050.900275] LNet: 104436:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2050.904853] LNetError: 104436:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2050.917774] LNet: Removed LNI 192.168.204.26@tcp [ 2051.661467] Key type .llcrypt unregistered [ 2051.664272] Key type ._llcrypt unregistered [ 2063.324594] Key type ._llcrypt registered [ 2063.328297] Key type .llcrypt registered [ 2064.928250] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 06:50:46 (1787136646) [ 2065.821291] Key type .llcrypt unregistered [ 2065.824847] Key type ._llcrypt unregistered [ 2072.219549] Key type ._llcrypt registered [ 2072.221233] Key type .llcrypt registered [ 2072.453225] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2072.482976] alg: No test for adler32 (adler32-zlib) [ 2079.762313] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 06:51:01 (1787136661) [ 2079.877792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2079.920203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2079.982983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2080.019439] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 2080.745523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2080.800255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2080.844908] LNet: Removed LNI 192.168.204.26@tcp [ 2081.194253] LNet: 106192:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2081.664841] Key type .llcrypt unregistered [ 2081.670209] Key type ._llcrypt unregistered [ 2094.651735] Key type ._llcrypt registered [ 2094.654269] Key type .llcrypt registered [ 2096.036597] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 06:51:17 (1787136677) [ 2096.421280] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2096.439392] alg: No test for adler32 (adler32-zlib) [ 2097.386704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2097.460531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2097.495313] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 2098.426196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.26@tcp [ 2098.510962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.204.26@tcp [ 2098.533912] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2098.537052] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2098.542538] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2098, 0, 0 [ 2098.548477] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2098.550632] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2098.560659] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2098.565545] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2098, 0, 0 [ 2098.572213] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2098.575778] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2098.583179] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2098.587761] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2098, 0, 2148 [ 2098.593508] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2098.600914] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2098.603960] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2098.610295] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2098, 0, 2148 [ 2098.616543] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2105.915597] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 06:51:27 (1787136687) [ 2106.542421] LNet: 107731:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2106.549421] LNetError: 107731:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2106.569207] LNet: Removed LNI 192.168.204.26@tcp [ 2107.411596] Key type .llcrypt unregistered [ 2107.414273] Key type ._llcrypt unregistered [ 2115.240197] Key type ._llcrypt registered [ 2115.243201] Key type .llcrypt registered [ 2115.394090] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2115.431035] alg: No test for adler32 (adler32-zlib) [ 2116.409680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2116.449731] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 2124.528733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.126@tcp [ 2134.867236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2134.927318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2134.989815] LNet: Added LNI 192.168.204.26@tcp1 [8/256/0/180] [ 2136.111255] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2136.257529] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2136.262529] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2136.271912] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2138.654522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2138.683647] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2149.900718] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 06:52:11 (1787136731) [ 2150.298923] LNet: 109245:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2150.332194] LNetError: 109245:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2150.340229] LNet: Removed LNI 192.168.204.26@tcp [ 2151.392825] LNet: Removed LNI 192.168.204.26@tcp1 [ 2151.394700] LNetError: 109245:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2151.419486] LNetError: 109245:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2151.440637] LNet: Removed LNI 10.1.2.3@tcp99 [ 2152.121520] Key type .llcrypt unregistered [ 2152.123387] Key type ._llcrypt unregistered [ 2160.093318] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2160.099417] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2162.658428] Key type ._llcrypt registered [ 2162.663084] Key type .llcrypt registered [ 2162.790416] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2162.803206] alg: No test for adler32 (adler32-zlib) [ 2163.832622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2163.934762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2163.961448] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 2164.990385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2165.016975] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2167.757912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2167.819337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2167.858862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2167.895533] LNetError: 109995:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2167.936577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2168.017754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.26@tcp [ 2168.739679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2168.898138] LNet: 110015:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2168.907895] LNetError: 110015:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2168.916294] LNet: Removed LNI 10.1.2.3@tcp [ 2169.959827] LNet: Removed LNI 192.168.204.26@tcp [ 2170.914576] Key type .llcrypt unregistered [ 2170.918179] Key type ._llcrypt unregistered [ 2183.128721] Key type ._llcrypt registered [ 2183.131873] Key type .llcrypt registered [ 2184.606627] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 06:52:46 (1787136766) [ 2185.002979] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2185.030805] alg: No test for adler32 (adler32-zlib) [ 2186.012098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2186.076674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2186.108634] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 2187.049108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.26@tcp [ 2187.144621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg426-client@tcp [ 2194.905912] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 06:52:56 (1787136776) [ 2195.155780] LNet: 111551:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2195.168445] LNetError: 111551:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2195.191211] LNet: Removed LNI 192.168.204.26@tcp [ 2196.067623] Key type .llcrypt unregistered [ 2196.071631] Key type ._llcrypt unregistered [ 2204.135644] Key type ._llcrypt registered [ 2204.137861] Key type .llcrypt registered [ 2204.375636] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2204.398707] alg: No test for adler32 (adler32-zlib) [ 2205.389186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2205.419476] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 2205.445746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2212.425376] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 06:53:14 (1787136794) [ 2214.304854] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2216.323982] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 06:53:17 (1787136797) [ 2217.768806] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2219.486961] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 06:53:21 (1787136801) [ 2219.832938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2219.862988] LNet: 112975:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2219.881851] LNetError: 112975:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2219.895659] LNet: Removed LNI 192.168.204.26@tcp [ 2219.917350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2219.975211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2220.012174] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 2220.993922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2221.091755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2229.207633] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 06:53:30 (1787136810) [ 2229.352635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2229.390546] LNet: 113680:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2229.402623] LNetError: 113680:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2230.443317] LNet: Removed LNI 192.168.204.26@tcp [ 2230.523754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2230.623767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2230.660508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2231.201500] LNet: 113725:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2231.887617] Key type .llcrypt unregistered [ 2231.890714] Key type ._llcrypt unregistered [ 2247.066562] Key type ._llcrypt registered [ 2247.070868] Key type .llcrypt registered [ 2248.515346] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 06:53:49 (1787136829) [ 2248.863057] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2248.879621] alg: No test for adler32 (adler32-zlib) [ 2249.779207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2249.839492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2249.858132] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 2250.485185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2250.531424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.26@tcp [ 2256.456968] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 06:53:58 (1787136838) [ 2256.585809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2256.612825] LNet: 115239:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2256.624666] LNetError: 115239:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2256.659236] LNet: Removed LNI 192.168.204.26@tcp [ 2256.701363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2256.777943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2256.809927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2263.519879] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 06:54:05 (1787136845) [ 2263.695708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2263.748851] LNet: 115820:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2263.784242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2263.880905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2264.210928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2264.242248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2264.320402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2264.908840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2264.961226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2265.101607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2265.136702] LNet: 115858:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2265.171421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2265.267227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2265.924732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2265.964735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2266.055659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2266.703609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2266.766898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2274.153365] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 06:54:15 (1787136855) [ 2274.284358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2274.313847] LNet: 116455:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2274.347158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2274.437634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2274.475793] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 2275.337341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2275.366130] LNet: Added LNI 192.168.204.26@tcp2 [8/256/0/180] [ 2276.384557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2276.460937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.26@tcp [ 2276.703668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2276.736896] LNetError: 116738:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2276.744986] LNet: Removed LNI 192.168.204.26@tcp [ 2276.784578] LNet: Removed LNI 192.168.204.26@tcp2 [ 2276.852118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2276.909668] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 2276.914559] LNet: Added LNI 192.168.204.26@tcp2 [8/256/0/180] [ 2276.918958] LNet: 116744:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2277.407973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.26@tcp [ 2277.654666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2278.752323] LNet: Removed LNI 192.168.204.26@tcp [ 2278.754896] LNetError: 116781:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2278.770360] LNetError: 116781:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2278.808307] LNet: Removed LNI 192.168.204.26@tcp2 [ 2278.857278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2278.981137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2279.011037] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 2280.199796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2280.235978] LNet: Added LNI 192.168.204.26@tcp2 [8/256/0/180] [ 2281.341614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.26@tcp [ 2281.394900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2281.459618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.26@tcp [ 2281.710868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2281.733922] LNet: 117065:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2281.742488] LNet: 117065:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2281.749979] LNetError: 117065:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2281.756200] LNet: Removed LNI 192.168.204.26@tcp [ 2282.791349] LNet: Removed LNI 192.168.204.26@tcp2 [ 2282.845887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2282.952079] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 2282.958869] LNet: Added LNI 192.168.204.26@tcp2 [8/256/0/180] [ 2282.972223] LNet: 117071:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2283.744135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.26@tcp [ 2284.026346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2284.062918] LNet: Removed LNI 192.168.204.26@tcp [ 2285.097178] LNet: Removed LNI 192.168.204.26@tcp2 [ 2285.127537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2285.221560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2285.264554] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 2286.450672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2286.484490] LNet: Added LNI 192.168.204.26@tcp2 [8/256/0/180] [ 2287.565552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2287.639555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.26@tcp [ 2287.887665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2287.908915] LNetError: 117388:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2287.913420] LNetError: 117388:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2288.928353] LNet: Removed LNI 192.168.204.26@tcp [ 2288.953927] LNet: Removed LNI 192.168.204.26@tcp2 [ 2289.006737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2289.095731] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 2289.106493] LNet: Added LNI 192.168.204.26@tcp2 [8/256/0/180] [ 2289.114734] LNet: 117394:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2289.689664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.26@tcp [ 2289.875912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2289.890712] LNet: 117431:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2289.895280] LNet: 117431:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2290.912273] LNet: Removed LNI 192.168.204.26@tcp [ 2290.940995] LNet: Removed LNI 192.168.204.26@tcp2 [ 2291.019890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2291.086324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2291.113837] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 2292.030759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2292.083503] LNet: Added LNI 192.168.204.26@tcp2 [8/256/0/180] [ 2293.115549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.26@tcp [ 2293.170603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2293.259965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.26@tcp [ 2293.560612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2294.628099] LNet: Removed LNI 192.168.204.26@tcp [ 2295.656425] LNet: Removed LNI 192.168.204.26@tcp2 [ 2295.728115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2295.866665] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 2295.879721] LNet: Added LNI 192.168.204.26@tcp2 [8/256/0/180] [ 2295.889754] LNet: 117722:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2296.633539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.26@tcp [ 2296.794812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2296.811489] LNetError: 117759:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2296.817523] LNetError: 117759:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2296.825473] LNet: Removed LNI 192.168.204.26@tcp [ 2297.895876] LNet: Removed LNI 192.168.204.26@tcp2 [ 2297.963720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2298.073276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2298.095694] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 2299.052825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2299.104920] LNet: Added LNI 192.168.204.26@tcp2 [8/256/0/180] [ 2300.174995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2300.210130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2300.265440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.26@tcp [ 2300.697297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2301.728253] LNet: Removed LNI 192.168.204.26@tcp [ 2301.748983] LNet: Removed LNI 192.168.204.26@tcp2 [ 2301.774610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2301.844336] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 2301.851680] LNet: Added LNI 192.168.204.26@tcp2 [8/256/0/180] [ 2301.860519] LNet: 118062:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2302.486971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.26@tcp [ 2302.908159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2303.969828] LNet: Removed LNI 192.168.204.26@tcp [ 2303.981570] LNet: Removed LNI 192.168.204.26@tcp2 [ 2304.044680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2304.108488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2304.128737] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 2305.049258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2305.070521] LNet: Added LNI 192.168.204.26@tcp2 [8/256/0/180] [ 2306.148570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.26@tcp [ 2306.191966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2306.223848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2306.315571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.26@tcp [ 2306.821528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2306.846899] LNet: 118416:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2306.852512] LNet: 118416:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2307.872219] LNet: Removed LNI 192.168.204.26@tcp [ 2308.903244] LNet: Removed LNI 192.168.204.26@tcp2 [ 2308.956136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2309.017671] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 2309.029725] LNet: Added LNI 192.168.204.26@tcp2 [8/256/0/180] [ 2309.838400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.26@tcp [ 2317.651257] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2319.143919] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 06:55:01 (1787136901) [ 2319.336165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2319.362862] LNetError: 119178:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2319.372865] LNetError: 119178:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2319.378452] LNet: Removed LNI 192.168.204.26@tcp [ 2320.424856] LNet: Removed LNI 192.168.204.26@tcp2 [ 2320.507634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2320.958990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2320.988369] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 2321.818717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2321.847163] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2322.833620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2322.865824] LNet: Removed LNI 10.1.2.3@tcp [ 2322.882301] LNet: Removed LNI 192.168.204.26@tcp [ 2322.915114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2323.048793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2323.096680] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 2323.111343] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2323.733720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2323.769159] LNet: Removed LNI 192.168.204.26@tcp [ 2323.780819] LNet: Removed LNI 10.1.2.3@tcp [ 2330.717977] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 06:55:12 (1787136912) [ 2330.899616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2330.964168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2331.027964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2331.053129] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 2331.924961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2331.985145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.26@tcp [ 2332.048875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.26@tcp [ 2332.092625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.26@tcp [ 2332.132905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.26@tcp [ 2332.168421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.26@tcp [ 2332.221681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.26@tcp [ 2332.263191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.26@tcp [ 2332.305160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.26@tcp [ 2332.362977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.26@tcp [ 2332.410179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.26@tcp [ 2332.599388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2339.299182] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 06:55:21 (1787136921) [ 2339.425657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2339.452431] LNet: 120789:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2339.460824] LNet: 120789:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2340.518689] LNet: Removed LNI 192.168.204.26@tcp [ 2340.563385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2340.702993] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2340.772384] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2340.780056] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2340.787771] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2343.273875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2343.298039] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 2344.010873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2344.028818] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2346.226231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.26@tcp [ 2346.271833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2346.384519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.204.26@tcp -m PUT -e local_timeout -r 1 [ 2346.524903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2346.630994] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2346.634357] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2347.105710] LNetError: 120863:0:(lib-move.c:4801:lnet_parse()) 192.168.204.26@tcp, src 192.168.204.26@tcp: Dropping PUT (error -108 looking up sender) [ 2348.133472] LNet: Removed LNI 10.1.2.3@tcp [ 2349.166118] LNet: Removed LNI 192.168.204.26@tcp [ 2349.728361] Key type .llcrypt unregistered [ 2349.736260] Key type ._llcrypt unregistered [ 2363.435168] Key type ._llcrypt registered [ 2363.438310] Key type .llcrypt registered [ 2364.952181] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 06:55:46 (1787136946) [ 2365.401772] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2365.421709] alg: No test for adler32 (adler32-zlib) [ 2366.403907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2366.968945] LNet: 122178:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2367.687590] Key type .llcrypt unregistered [ 2367.691051] Key type ._llcrypt unregistered [ 2373.775220] Key type ._llcrypt registered [ 2373.777951] Key type .llcrypt registered [ 2373.959275] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2373.973227] alg: No test for adler32 (adler32-zlib) [ 2374.890517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2374.935670] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 2383.218833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.126@tcp [ 2383.425714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2384.065065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.126@tcp [ 2384.162934] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.126@tcp [ 2384.324881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2384.353331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2384.396699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2384.429931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2384.469419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2384.503959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2384.552128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2384.588389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2384.604826] LNet: 122632:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2384.620763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2384.654034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2384.685835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2384.712148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2384.737978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2384.763907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2385.561462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2385.884958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2385.937993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2385.980524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2386.020341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2386.079793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2386.145347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2386.210834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2386.271802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2386.298319] LNet: 122749:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2386.311622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2386.351357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2386.388461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2386.422972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2386.467311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2386.500268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2387.138736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.126@tcp [ 2387.432508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2387.470410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2387.496834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2387.533942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2387.567305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2387.608957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2387.647988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2387.682732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2387.698021] LNet: 122866:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2387.714748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2387.757719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2387.796399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2387.839726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2387.893249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2387.934698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2388.518119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2388.731678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2388.756970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2388.781428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2388.806307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2388.838501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2388.858844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2388.884417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2388.912741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2388.945747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2388.972235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2388.996815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2389.022427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2389.050555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2389.080906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2389.409687] LNet: 123033:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2389.414847] LNetError: 123033:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2389.430697] LNet: Removed LNI 192.168.204.26@tcp [ 2389.911898] Key type .llcrypt unregistered [ 2389.914522] Key type ._llcrypt unregistered [ 2405.480670] Key type ._llcrypt registered [ 2405.482652] Key type .llcrypt registered [ 2406.732824] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 06:56:28 (1787136988) [ 2406.998517] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2407.016434] alg: No test for adler32 (adler32-zlib) [ 2407.980841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2408.424685] LNet: 123920:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2408.992997] Key type .llcrypt unregistered [ 2408.999739] Key type ._llcrypt unregistered [ 2415.830708] Key type ._llcrypt registered [ 2415.833659] Key type .llcrypt registered [ 2416.050592] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2416.061893] alg: No test for adler32 (adler32-zlib) [ 2416.967941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2416.996521] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 2424.057159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.126@tcp [ 2431.588602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2431.640866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2431.664971] LNet: Added LNI 192.168.204.26@tcp1 [8/256/0/180] [ 2432.651459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2433.063489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.126@tcp [ 2433.243041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2433.263672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2433.289308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2433.322517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2433.353641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2433.376229] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.126@tcp [ 2433.393325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2433.424149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2433.449869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2433.468438] LNet: 124576:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2433.487317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2433.539236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2433.616533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2433.652836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2433.705389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2433.769418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2434.305551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2434.496583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2434.521412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2434.545531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2434.580034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2434.603607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2434.632978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2434.665930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2434.705364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2434.720607] LNet: 124693:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2434.745175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2434.773791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2434.808199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2434.839741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2434.872882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2434.905933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2435.494105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.126@tcp [ 2435.754502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2435.781634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2435.803747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2435.824719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2435.853868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2435.876929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2435.906518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2435.944583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2435.960957] LNet: 124810:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2435.986630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2436.040025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2436.068413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2436.096976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2436.135736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2436.162547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2436.884898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2437.132782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2437.158781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2437.189632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2437.216692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2437.247274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2437.273916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2437.313151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2437.345428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2437.378439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2437.408673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2437.442678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2437.482039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2437.526530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2437.592422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2442.967519] LNet: 125012:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2442.983122] LNetError: 125012:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2443.003218] LNet: Removed LNI 192.168.204.26@tcp [ 2443.018808] LNet: Removed LNI 192.168.204.26@tcp1 [ 2443.484692] Key type .llcrypt unregistered [ 2443.493680] Key type ._llcrypt unregistered [ 2462.169766] Key type ._llcrypt registered [ 2462.171986] Key type .llcrypt registered [ 2463.468083] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 06:57:25 (1787137045) [ 2463.845895] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2463.868792] alg: No test for adler32 (adler32-zlib) [ 2464.780762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2464.851783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2464.876827] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 2465.866050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2465.907027] LNet: Added LNI 192.168.204.26@tcp2 [8/256/0/180] [ 2466.724777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.26@tcp [ 2466.824748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2467.875464] LNet: Removed LNI 192.168.204.26@tcp [ 2474.211434] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 06:57:36 (1787137056) [ 2475.666331] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2477.518946] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 06:57:39 (1787137059) [ 2484.987153] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 06:57:46 (1787137066) [ 2493.178106] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 06:57:54 (1787137074) [ 2501.290759] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 06:58:03 (1787137083) [ 2508.267560] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 06:58:10 (1787137090) [ 2516.750553] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 06:58:18 (1787137098) [ 2525.434232] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 06:58:26 (1787137106) [ 2533.601526] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 06:58:35 (1787137115) [ 2543.447696] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 06:58:44 (1787137124) [ 2551.596418] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 06:58:53 (1787137133) [ 2558.984150] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 06:59:01 (1787137141) [ 2566.069481] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 06:59:07 (1787137147) [ 2573.379317] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 06:59:15 (1787137155) [ 2580.731795] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 06:59:22 (1787137162) [ 2587.996081] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 06:59:29 (1787137169) [ 2594.953181] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 06:59:36 (1787137176) [ 2602.771322] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 06:59:44 (1787137184) [ 2610.074055] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 06:59:51 (1787137191) [ 2610.220868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2610.279536] LNet: 136704:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2610.294533] LNetError: 136704:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2610.315780] LNet: Removed LNI 192.168.204.26@tcp2 [ 2610.369476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2610.501948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2610.521853] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 2611.747713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2611.764990] LNet: 136851:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2611.770313] LNetError: 136851:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2611.784398] LNet: Removed LNI 192.168.204.26@tcp [ 2611.826549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2611.884271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2611.905575] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 2617.916778] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 06:59:59 (1787137199) [ 2625.510170] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 07:00:07 (1787137207) [ 2632.556273] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 07:00:14 (1787137214) [ 2639.524478] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 07:00:21 (1787137221) [ 2646.371705] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 07:00:28 (1787137228) [ 2654.832775] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 07:00:36 (1787137236) [ 2662.398209] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 07:00:44 (1787137244) [ 2669.437751] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 07:00:51 (1787137251) [ 2677.076970] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 07:00:58 (1787137258) [ 2685.281489] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 07:01:06 (1787137266) [ 2692.994569] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 07:01:14 (1787137274) [ 2699.983452] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 07:01:21 (1787137281) [ 2706.750698] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2431 sec ========== 07:01:28 (1787137288) [ 2707.141848] LNet: 144331:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2707.150373] LNetError: 144331:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2707.185142] LNet: Removed LNI 192.168.204.26@tcp [ 2707.961405] Key type .llcrypt unregistered [ 2707.965375] Key type ._llcrypt unregistered [ 2721.171264] Key type ._llcrypt registered [ 2721.176726] Key type .llcrypt registered [ 2721.598832] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2721.611041] alg: No test for adler32 (adler32-zlib) [ 2722.736758] Lustre: Lustre: Build Version: 2.17.57_11_g9d7c9d8 [ 2723.041048] LNet: Added LNI 192.168.204.26@tcp [8/256/0/180] [ 2724.736917] Key type lgssc registered [ 2725.894349] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2787.040454] Lustre: Mounted lustre-client - version 2.17.57_11_g9d7c9d8 [ 2792.168511] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2799.417130] Lustre: Unmounted lustre-client [ 2817.071739] Key type lgssc unregistered [ 2817.352394] LNet: 147289:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2817.366092] LNetError: 147289:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2817.386341] LNet: Removed LNI 192.168.204.26@tcp [ 2818.129431] Key type .llcrypt unregistered [ 2818.136483] Key type ._llcrypt unregistered