[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 430495647 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.996 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.002364] x2apic enabled [ 0.003011] Switched APIC routing to physical x2apic. [ 0.004017] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229833f6470, max_idle_ns: 440795327230 ns [ 0.007021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399996) [ 0.009003] pid_max: default: 32768 minimum: 301 [ 0.010139] LSM: Security Framework initializing [ 0.011057] Yama: becoming mindful. [ 0.012040] SELinux: Initializing. [ 0.013082] *** VALIDATE selinux *** [ 0.021853] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026181] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027155] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028104] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029131] *** VALIDATE tmpfs *** [ 0.031197] *** VALIDATE proc *** [ 0.032223] *** VALIDATE cgroup *** [ 0.033012] *** VALIDATE cgroup2 *** [ 0.034271] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035154] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037029] Spectre V2 : User space: Vulnerable [ 0.038012] Speculative Store Bypass: Vulnerable [ 0.040785] debug: unmapping init [mem 0xffffffffb3059000-0xffffffffb3060fff] [ 0.042146] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043742] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044024] ... version: 2 [ 0.044985] ... bit width: 48 [ 0.045015] ... generic registers: 4 [ 0.046013] ... value mask: 0000ffffffffffff [ 0.047012] ... max period: 00007fffffffffff [ 0.048016] ... fixed-purpose events: 3 [ 0.048969] ... event mask: 000000070000000f [ 0.049304] rcu: Hierarchical SRCU implementation. [ 0.051329] smp: Bringing up secondary CPUs ... [ 0.052524] x86: Booting SMP configuration: [ 0.053023] .... node #0, CPUs: #1 #2 #3 [ 0.056208] smp: Brought up 1 node, 4 CPUs [ 0.058013] smpboot: Max logical packages: 1 [ 0.059018] smpboot: Total of 4 processors activated (19199.96 BogoMIPS) [ 0.145352] node 0 deferred pages initialised in 84ms [ 0.148167] devtmpfs: initialized [ 0.150202] x86/mm: Memory block size: 128MB [ 0.152797] gcov: version magic: 0x41383552 [ 0.156277] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.159139] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.162360] pinctrl core: initialized pinctrl subsystem [ 0.164214] [ 0.164779] ************************************************************* [ 0.167012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.169011] ** ** [ 0.171014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.173014] ** ** [ 0.176013] ** This means that this kernel is built to expose internal ** [ 0.178010] ** IOMMU data structures, which may compromise security on ** [ 0.180015] ** your system. ** [ 0.183012] ** ** [ 0.185012] ** If you see this message and you are not debugging the ** [ 0.187013] ** kernel, report this immediately to your vendor! ** [ 0.189013] ** ** [ 0.192013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.194010] ************************************************************* [ 0.196697] NET: Registered protocol family 16 [ 0.198423] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.201057] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.204066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.208032] cpuidle: using governor menu [ 0.210040] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.211588] PCI: Using configuration type 1 for base access [ 0.213139] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.225056] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.227023] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.232056] cryptd: max_cpu_qlen set to 1000 [ 0.236278] ACPI: Added _OSI(Module Device) [ 0.238014] ACPI: Added _OSI(Processor Device) [ 0.240016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.241011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.245143] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.251630] ACPI: Interpreter enabled [ 0.253058] ACPI: PM: (supports S0 S3 S4 S5) [ 0.255014] ACPI: Using IOAPIC for interrupt routing [ 0.256110] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.260511] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.271766] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.274040] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.276025] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.280108] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.285449] acpiphp: Slot [2] registered [ 0.287110] acpiphp: Slot [5] registered [ 0.288124] acpiphp: Slot [6] registered [ 0.290105] acpiphp: Slot [3] registered [ 0.291101] acpiphp: Slot [4] registered [ 0.293210] acpiphp: Slot [7] registered [ 0.294094] acpiphp: Slot [8] registered [ 0.296105] acpiphp: Slot [9] registered [ 0.297000] acpiphp: Slot [10] registered [ 0.297000] acpiphp: Slot [11] registered [ 0.299096] acpiphp: Slot [12] registered [ 0.301101] acpiphp: Slot [13] registered [ 0.302075] acpiphp: Slot [14] registered [ 0.303090] acpiphp: Slot [15] registered [ 0.305075] acpiphp: Slot [16] registered [ 0.306054] acpiphp: Slot [17] registered [ 0.307035] acpiphp: Slot [18] registered [ 0.308026] acpiphp: Slot [19] registered [ 0.309079] acpiphp: Slot [20] registered [ 0.310078] acpiphp: Slot [21] registered [ 0.311000] acpiphp: Slot [22] registered [ 0.311000] acpiphp: Slot [23] registered [ 0.313088] acpiphp: Slot [24] registered [ 0.314117] acpiphp: Slot [25] registered [ 0.316086] acpiphp: Slot [26] registered [ 0.317079] acpiphp: Slot [27] registered [ 0.318079] acpiphp: Slot [28] registered [ 0.319071] acpiphp: Slot [29] registered [ 0.320077] acpiphp: Slot [30] registered [ 0.321112] acpiphp: Slot [31] registered [ 0.322058] PCI host bridge to bus 0000:00 [ 0.322989] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.325015] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.326030] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.328016] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.330019] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.332018] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.333144] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.335019] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.337563] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.343651] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.346505] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.349016] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.351019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.353022] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.355475] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.357778] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.360050] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.363846] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.366797] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.375017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.379015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.384455] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.391020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.398016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.415016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.426568] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.432017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.439025] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.455018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.468304] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.471385] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.473387] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.475282] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.478271] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.482279] iommu: Default domain type: Passthrough [ 0.484615] SCSI subsystem initialized [ 0.486247] ACPI: bus type USB registered [ 0.488196] usbcore: registered new interface driver usbfs [ 0.490164] usbcore: registered new interface driver hub [ 0.492136] usbcore: registered new device driver usb [ 0.494214] pps_core: LinuxPPS API ver. 1 registered [ 0.496022] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.500065] PTP clock support registered [ 0.502138] EDAC MC: Ver: 3.0.0 [ 0.503452] PCI: Using ACPI for IRQ routing [ 0.504684] NetLabel: Initializing [ 0.505008] NetLabel: domain hash size = 128 [ 0.506013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.507109] NetLabel: unlabeled traffic allowed by default [ 0.509132] vgaarb: loaded [ 0.510216] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.512012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.517377] clocksource: Switched to clocksource kvm-clock [ 0.629510] VFS: Disk quotas dquot_6.6.0 [ 0.631421] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.634211] *** VALIDATE ramfs *** [ 0.635572] *** VALIDATE hugetlbfs *** [ 0.637410] pnp: PnP ACPI init [ 0.640150] pnp: PnP ACPI: found 6 devices [ 0.660388] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.663654] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.665791] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.667733] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.670293] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.672669] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.675296] NET: Registered protocol family 2 [ 0.677940] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.682793] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.686296] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.691882] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.695760] TCP: Hash tables configured (established 65536 bind 65536) [ 0.699149] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.702457] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.705645] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.708960] NET: Registered protocol family 1 [ 0.712310] RPC: Registered named UNIX socket transport module. [ 0.713868] RPC: Registered udp transport module. [ 0.715386] RPC: Registered tcp transport module. [ 0.716840] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.719058] NET: Registered protocol family 44 [ 0.720687] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.722695] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.724677] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.726452] PCI: CLS 0 bytes, default 64 [ 0.727654] Unpacking initramfs... [ 2.126072] debug: unmapping init [mem 0xffff9d1cbcc64000-0xffff9d1cbffcffff] [ 2.129838] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.132174] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.136131] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229833f6470, max_idle_ns: 440795327230 ns [ 2.621745] Initialise system trusted keyrings [ 2.623470] Key type blacklist registered [ 2.625603] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.634675] zbud: loaded [ 2.637838] *** VALIDATE nfs *** [ 2.639028] *** VALIDATE nfs4 *** [ 2.640735] pstore: using deflate compression [ 2.644197] Platform Keyring initialized [ 2.752875] NET: Registered protocol family 38 [ 2.754951] Key type asymmetric registered [ 2.756477] Asymmetric key parser 'x509' registered [ 2.758720] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.762496] io scheduler mq-deadline registered [ 2.764035] io scheduler kyber registered [ 2.765595] io scheduler bfq registered [ 2.767615] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.771315] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.774528] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.777924] ACPI: Power Button [PWRF] [ 2.783859] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.791612] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.802370] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.831287] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.860679] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.866100] Non-volatile memory driver v1.3 [ 2.868041] Linux agpgart interface v0.103 [ 2.902698] virtio_blk virtio1: [vda] 144824 512-byte logical blocks (74.1 MB/70.7 MiB) [ 2.905897] vda: detected capacity change from 0 to 74149888 [ 2.921981] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.925043] vdb: detected capacity change from 0 to 1073741824 [ 2.931568] libphy: Fixed MDIO Bus: probed [ 2.940610] usbcore: registered new interface driver usbserial_generic [ 2.942480] usbserial: USB Serial support registered for generic [ 2.944453] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.948578] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.950712] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.953109] mousedev: PS/2 mouse device common for all mice [ 2.955394] rtc_cmos 00:05: RTC can wake from S4 [ 2.957334] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.960408] rtc_cmos 00:05: registered as rtc0 [ 2.962949] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.964041] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.966568] intel_pstate: CPU model not supported [ 2.969414] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.972689] hid: raw HID events driver (C) Jiri Kosina [ 2.974898] usbcore: registered new interface driver usbhid [ 2.977203] usbhid: USB HID core driver [ 2.979172] drop_monitor: Initializing network drop monitor service [ 2.982354] Initializing XFRM netlink socket [ 2.984528] NET: Registered protocol family 10 [ 2.987895] Segment Routing with IPv6 [ 2.989506] NET: Registered protocol family 17 [ 2.991645] mpls_gso: MPLS GSO support [ 2.999391] RAS: Correctable Errors collector initialized. [ 3.002461] AVX version of gcm_enc/dec engaged. [ 3.004438] AES CTR mode by8 optimization enabled [ 3.074200] sched_clock: Marking stable (3074175063, 0)->(3918981475, -844806412) [ 3.076480] registered taskstats version 1 [ 3.078410] Loading compiled-in X.509 certificates [ 3.079742] zswap: loaded using pool lzo/zbud [ 3.104093] Key type big_key registered [ 3.117480] Key type encrypted registered [ 3.119184] ima: No TPM chip found, activating TPM-bypass! [ 3.121275] ima: Allocated hash algorithm: sha1 [ 3.122880] ima: No architecture policies found [ 3.124712] evm: Initialising EVM extended attributes: [ 3.126702] evm: security.selinux [ 3.128237] evm: security.ima [ 3.129323] evm: security.capability [ 3.130634] evm: HMAC attrs: 0x1 [ 3.133235] rtc_cmos 00:05: setting system clock to 2026-06-27 03:16:12 UTC (1782530172) [ 3.139607] debug: unmapping init [mem 0xffffffffb4003000-0xffffffffb41fffff] [ 3.142217] debug: unmapping init [mem 0xffffffffb2d82000-0xffffffffb3058fff] [ 3.151097] Write protecting the kernel read-only data: 28672k [ 3.154493] debug: unmapping init [mem 0xffffffffb1403000-0xffffffffb15fffff] [ 3.156129] debug: unmapping init [mem 0xffffffffb1d14000-0xffffffffb1dfffff] [ 3.187598] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.196037] systemd[1]: Detected virtualization kvm. [ 3.197438] systemd[1]: Detected architecture x86-64. [ 3.198807] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.226598] systemd[1]: No hostname configured. [ 3.227871] systemd[1]: Set hostname to . [ 3.229989] random: systemd: uninitialized urandom read (16 bytes read) [ 3.232037] systemd[1]: Initializing machine ID from random generator. [ 3.272708] random: ln: uninitialized urandom read (6 bytes read) [ 3.382401] random: systemd: uninitialized urandom read (16 bytes read) [ 3.385311] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.389700] systemd[1]: Reached target Paths. [ OK ] Reached target Paths. [ 3.392804] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Slices. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Reached target Swap. [ OK ] Reached target Timers. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.086772] device-mapper: uevent: version 1.0.3 [ 4.089107] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.897075] virtio_net virtio0 ens2: renamed from eth0 [ 5.037306] scsi host0: ata_piix [ 5.054665] scsi host1: ata_piix [ 5.056244] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.058491] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.895695] dracut-initqueue[584]: RTNETLINK answers: File exists [ 9.639399] random: crng init done [ 9.641029] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.217390] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.438533] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.690805] SELinux: Disabled at runtime. [ 11.756539] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.764483] systemd[1]: Detected virtualization kvm. [ 11.766539] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.229598] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.232846] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.242484] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.246726] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.250284] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.258121] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.269271] systemd[1]: Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Remount Root and Kernel File Systems... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice User and Session Slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target Slices. [ OK ] Created slice system-getty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Huge Pages File System... [ OK ] Reached target rpc_pipefs.target. [ 12.398722] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. Mounting Kernel Debug File System... [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. Mounting POSIX Message Queue File System... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Flush Journal to Persistent Storage. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 12.681150] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 12.913884] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.927072] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.011341] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.024811] EDAC sbridge: Ver: 1.1.2 [ 14.170878] Key type dns_resolver registered [ 14.469971] NFS: Registering the id_resolver key type [ 14.472447] Key type id_resolver registered [ 14.474415] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg227-client login: [ 74.299601] libcfs: loading out-of-tree module taints kernel. [ 74.448666] Key type ._llcrypt registered [ 74.467068] Key type .llcrypt registered [ 75.261680] hrtimer: interrupt took 5782228 ns [ 75.326749] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 75.343963] alg: No test for adler32 (adler32-zlib) [ 77.015756] Lustre: Lustre: Build Version: 2.17.54_99_gde77dd7 [ 78.605147] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 80.496247] Key type lgssc registered [ 82.980412] Lustre: Echo OBD driver; http://www.lustre.org/ [ 221.804888] Lustre: Mounted lustre-client [ 227.698628] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 246.750675] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing check_logdir /tmp/testlogs/ [ 247.265952] Lustre: lustre-OST0000-osc-ffff9d1d08154000: disconnect after 23s idle [ 252.823798] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing yml_node [ 257.498436] Lustre: DEBUG MARKER: Client: 2.17.54.99 [ 260.621533] Lustre: DEBUG MARKER: MDS: 2.17.54.99 [ 263.822761] Lustre: DEBUG MARKER: OSS: 2.17.54.99 [ 265.912909] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Fri Jun 26 23:20:33 EDT 2026 [ 283.673612] Lustre: Unmounted lustre-client [ 305.206724] Key type lgssc unregistered [ 305.668887] LNet: 8417:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 305.677556] LNetError: 8417:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 305.705421] LNet: Removed LNI 192.168.202.27@tcp [ 306.927345] Key type .llcrypt unregistered [ 306.934940] Key type ._llcrypt unregistered [ 337.255175] Key type ._llcrypt registered [ 337.257633] Key type .llcrypt registered [ 337.313859] LNet: 8997:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 337.313859] mask usage: [+|-] ... [ 337.686835] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 337.719541] alg: No test for adler32 (adler32-zlib) [ 338.713165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 338.770821] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 338.792761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 343.362204] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 343.589095] LNet: 9302:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 343.595160] LNetError: 9302:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 343.608284] LNet: Removed LNI 192.168.202.27@tcp [ 344.315714] Key type .llcrypt unregistered [ 344.317373] Key type ._llcrypt unregistered [ 352.880681] Key type ._llcrypt registered [ 352.886503] Key type .llcrypt registered [ 354.511504] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 23:22:02 (1782530522) [ 354.815092] LNet: 9676:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 354.815092] mask usage: [+|-] ... [ 354.964947] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 354.983909] alg: No test for adler32 (adler32-zlib) [ 355.933838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 356.023179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 356.075237] LNet: 9714:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 356.126680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 356.191846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 356.211616] LNet: 9721:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 364.838515] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 23:22:12 (1782530532) [ 365.260675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 365.316937] LNet: 10301:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 365.354620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 365.489445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 374.774408] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 23:22:22 (1782530542) [ 374.949623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 374.981902] LNet: 10881:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 375.005475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 375.122460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 383.790800] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 23:22:31 (1782530551) [ 383.925675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 383.971887] LNet: 11465:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 384.015140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 384.141909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 392.074479] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 23:22:40 (1782530560) [ 392.484809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 392.504445] LNet: 12058:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 392.536303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 392.665393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 393.250416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 400.888550] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 23:22:48 (1782530568) [ 401.044737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 401.088628] LNet: 12643:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 401.121444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 401.342653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 401.887528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 402.302952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 410.656852] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 23:22:58 (1782530578) [ 410.834164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 410.899947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 411.004967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 419.815738] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 23:23:07 (1782530587) [ 419.928465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 419.988323] LNet: 13823:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 419.995560] LNet: 13823:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 420.040929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 420.141885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 420.191154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 420.769678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 420.808484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 421.509524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 421.548789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 422.117193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 422.178613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 422.841324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 422.902029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 423.570737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 423.645540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 424.297357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 424.361077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 425.092477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 425.148155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 425.747555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 425.824991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 426.723502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 426.826430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 427.347913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 427.411075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 428.025439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 428.127637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 428.738700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 428.852761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 438.144423] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 23:23:25 (1782530605) [ 438.388377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 438.478837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 438.589717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 438.644987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 446.630971] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 23:23:34 (1782530614) [ 446.832253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 446.910860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 447.031761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 447.063935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 455.465616] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 23:23:43 (1782530623) [ 455.692867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 455.755076] LNet: 15681:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 455.763868] LNet: 15681:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 455.819505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 455.972180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 456.064875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 465.163562] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 23:23:52 (1782530632) [ 465.282452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 465.373195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 465.507088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 465.585759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 475.839792] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 23:24:03 (1782530643) [ 476.068329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 476.169788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 476.271678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 476.326996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 485.934063] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 23:24:13 (1782530653) [ 486.076608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 486.147670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 486.270666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 486.379668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 495.185143] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 23:24:23 (1782530663) [ 495.678646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 495.773845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 495.892442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 496.317858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 504.074124] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 23:24:31 (1782530671) [ 504.304778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 504.363402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 504.641689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 504.995590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 512.002778] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 23:24:39 (1782530679) [ 512.145446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 512.201724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 512.403097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 512.592916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 522.028254] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 23:24:49 (1782530689) [ 522.272734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 522.329184] LNet: 19859:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 522.338507] LNet: 19859:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 6 previous similar messages [ 522.384647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 523.048531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 523.406661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 532.104263] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 23:24:59 (1782530699) [ 532.427357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 532.486421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 532.598358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 532.670242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 540.140791] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 23:25:07 (1782530707) [ 540.479972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 540.542759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 540.638410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 540.731536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 549.143319] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 23:25:16 (1782530716) [ 549.420551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 549.458052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 549.561436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 558.504247] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 23:25:26 (1782530726) [ 558.610941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 558.681194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 558.801972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 567.067245] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 23:25:34 (1782530734) [ 567.202143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 567.251690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 567.415595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 568.028860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 575.568593] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 23:25:43 (1782530743) [ 575.870654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 575.919461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 576.017763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 576.092058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 584.886585] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 23:25:52 (1782530752) [ 585.055441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 585.121123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 585.273443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 585.336865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 593.230724] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 23:26:00 (1782530760) [ 593.358856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 593.463188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 593.614505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 593.669871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 603.379688] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 23:26:11 (1782530771) [ 603.588881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 603.697039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 603.831696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 603.860907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 613.420948] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 23:26:21 (1782530781) [ 613.479572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 613.511699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 613.581768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 613.650217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 613.678855] LNetError: 25837:0:(peer.c:2234:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 613.728810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 613.763778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 620.087967] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 23:26:28 (1782530788) [ 620.260671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 620.325957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 620.402036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.XMg7J4pgGq [ 620.464385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.XMg7J4pgGq [ 620.521176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.XMg7J4pgGq [ 626.832845] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 23:26:34 (1782530794) [ 626.978178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 627.024169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 627.073344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 627.097741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 627.133728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 634.945553] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 23:26:42 (1782530802) [ 635.176589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 635.224985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 635.302518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 638.288561] LNet: Message forwarding will be enabled in 2s [ 641.376225] LNet: Message forwarding enabled [ 641.823859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 641.872186] LNet: Message forwarding will stop in 2s to 6s [ 646.412202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 646.833234] LNet: Message forwarding disabled [ 648.665216] LNet: Message forwarding will be enabled in 2s [ 652.000306] LNet: Message forwarding enabled [ 656.256550] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 23:27:04 (1782530824) [ 656.661219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 656.698780] LNet: 28284:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 656.704406] LNet: 28284:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 13 previous similar messages [ 658.280271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 658.393994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 665.037123] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 23:27:13 (1782530833) [ 665.201351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 665.252925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 665.358951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 665.417636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 665.475454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 665.540953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 665.580670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 665.620264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 665.661433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 665.716883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 665.753315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 665.799961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 665.846388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 665.883069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 665.956929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 666.030354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 666.078136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 666.136583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 666.200944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 666.261248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 666.332493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 666.357854] LNetError: 28909:0:(router.c:624:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 673.785383] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 23:27:21 (1782530841) [ 674.062699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 674.255630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 674.391234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 682.859883] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 23:27:30 (1782530850) [ 684.788946] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg227-server [ 686.704223] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 23:27:34 (1782530854) [ 687.001865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 687.039619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 687.101880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 687.140147] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 688.335571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.202.[28-32]@tcp [ 688.374732] LNet: 30628:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 688.447204] LNetError: 30417:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.28@tcp is being used as a gateway but routing feature is not turned on [ 688.470378] LNetError: 30417:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp1 through 192.168.202.28@tcp has gone from up to down [ 689.445022] LNetError: 30417:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.30@tcp is being used as a gateway but routing feature is not turned on [ 689.456276] LNetError: 30417:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 7 previous similar messages [ 690.487042] LNetError: 30417:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.29@tcp is being used as a gateway but routing feature is not turned on [ 690.495067] LNetError: 30417:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 5 previous similar messages [ 692.534054] LNetError: 30417:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.29@tcp is being used as a gateway but routing feature is not turned on [ 692.560503] LNetError: 30417:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 696.040196] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 23:27:43 (1782530863) [ 696.611992] LNetError: 30417:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.28@tcp is being used as a gateway but routing feature is not turned on [ 696.626972] LNetError: 30417:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 698.068523] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg227-server [ 699.881936] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 23:27:48 (1782530868) [ 700.108895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 700.160219] LNetError: 31563:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 700.195312] LNet: Removed LNI 192.168.202.27@tcp [ 700.234709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 700.312640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 700.349753] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 700.953492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.202.[28-33/2]@tcp [ 700.969850] LNet: 31682:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 700.975906] LNet: 31682:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 700.990661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.202.[28-33/2]@tcp [ 701.021428] LNetError: 31568:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp103 through 192.168.202.28@tcp has gone from up to down [ 701.033195] LNetError: 31568:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 707.444857] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 23:27:55 (1782530875) [ 707.601841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 707.650371] LNetError: 32267:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 708.715183] LNet: Removed LNI 192.168.202.27@tcp [ 708.777050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 708.963155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 709.089601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 709.167562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 709.350192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 709.446957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 709.515987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 709.633847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 709.788204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 709.913150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 710.108952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 710.282358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 710.375147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 710.531916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 710.695388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 710.751986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 710.878944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 711.002289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 711.115362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 711.231703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 711.409562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 711.485369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 711.587896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 711.692947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 711.752122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 711.862858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 712.006268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 712.087774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 712.264922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 712.396270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 712.451888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 712.530377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 712.638342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 712.720606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 712.849646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 720.552723] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 23:28:07 (1782530887) [ 723.315738] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg227-server [ 725.598662] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 23:28:13 (1782530893) [ 728.340834] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg227-server [ 731.161479] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 23:28:18 (1782530898) [ 732.277793] Key type .llcrypt unregistered [ 732.279591] Key type ._llcrypt unregistered [ 743.546585] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 743.987954] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 744.059162] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 744.078069] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 746.977266] Key type ._llcrypt registered [ 747.000454] Key type .llcrypt registered [ 747.086454] LNet: 33957:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 747.086454] mask usage: [+|-] ... [ 747.517258] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 747.546124] alg: No test for adler32 (adler32-zlib) [ 748.621454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 748.797466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 748.866391] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 750.239407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 750.307551] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 752.908294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 752.929530] LNet: Removed LNI 10.1.2.3@tcp [ 753.036769] LNet: 34400:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 753.043050] LNetError: 34400:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 753.057933] LNet: Removed LNI 192.168.202.27@tcp [ 753.622874] Key type .llcrypt unregistered [ 753.625960] Key type ._llcrypt unregistered [ 768.121553] Key type ._llcrypt registered [ 768.123825] Key type .llcrypt registered [ 769.395770] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 23:28:57 (1782530937) [ 769.925454] Key type .llcrypt unregistered [ 769.929008] Key type ._llcrypt unregistered [ 776.085854] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 776.165178] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 776.172042] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 776.181025] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 778.807118] Key type ._llcrypt registered [ 778.808793] Key type .llcrypt registered [ 778.839249] LNet: 35436:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 778.839249] mask usage: [+|-] ... [ 779.003584] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 779.018527] alg: No test for adler32 (adler32-zlib) [ 779.904208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 779.955815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 779.992392] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 780.797486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 780.813381] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 783.293351] LNet: 35894:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 783.299699] LNetError: 35894:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 783.306380] LNet: Removed LNI 10.1.2.3@tcp [ 783.318899] LNet: Removed LNI 192.168.202.27@tcp [ 783.966968] Key type .llcrypt unregistered [ 783.972414] Key type ._llcrypt unregistered [ 797.423946] Key type ._llcrypt registered [ 797.428968] Key type .llcrypt registered [ 799.760359] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 23:29:26 (1782530966) [ 800.677466] Key type .llcrypt unregistered [ 800.678815] Key type ._llcrypt unregistered [ 809.419624] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 809.625767] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 809.632853] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 809.637808] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 812.549189] Key type ._llcrypt registered [ 812.561462] Key type .llcrypt registered [ 812.637067] LNet: 36925:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 812.637067] mask usage: [+|-] ... [ 812.818874] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 812.841556] alg: No test for adler32 (adler32-zlib) [ 813.793125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 813.831147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 813.861592] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 815.924573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 815.954490] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 818.382035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 818.407884] LNet: Removed LNI 10.1.2.3@tcp [ 818.423756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 818.451186] LNet: Removed LNI 10.1.2.31@tcp [ 818.467819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 818.486285] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 819.404732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 819.419832] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 820.341223] LNet: 37808:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 820.353657] LNetError: 37808:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 820.363137] LNet: Removed LNI 10.1.2.31@tcp [ 820.379917] LNet: Removed LNI 10.1.2.3@tcp [ 821.064361] Key type .llcrypt unregistered [ 821.067608] Key type ._llcrypt unregistered [ 836.763341] Key type ._llcrypt registered [ 836.769548] Key type .llcrypt registered [ 838.153211] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 23:30:05 (1782531005) [ 838.441791] LNet: 38600:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 838.441791] mask usage: [+|-] ... [ 838.629405] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 838.641199] alg: No test for adler32 (adler32-zlib) [ 839.652766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 839.742465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 839.797589] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 840.914037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 840.954581] LNet: 38761:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 840.967519] LNetError: 38761:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 840.986749] LNet: Removed LNI 192.168.202.27@tcp [ 841.022878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 841.065459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.202.27@tcp [ 841.092075] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 841.602465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.202.27@tcp [ 841.626969] LNet: Removed LNI 192.168.202.27@tcp [ 841.673424] LNet: 38896:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 842.292394] Key type .llcrypt unregistered [ 842.294445] Key type ._llcrypt unregistered [ 856.592766] Key type ._llcrypt registered [ 856.594716] Key type .llcrypt registered [ 858.341701] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 23:30:25 (1782531025) [ 860.641469] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg227-server [ 862.866765] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 23:30:30 (1782531030) [ 863.534435] Key type .llcrypt unregistered [ 863.536975] Key type ._llcrypt unregistered [ 871.019879] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 871.182446] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 871.187445] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 871.192677] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 873.896838] Key type ._llcrypt registered [ 873.901554] Key type .llcrypt registered [ 873.948126] LNet: 40274:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 873.948126] mask usage: [+|-] ... [ 874.162672] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 874.177246] alg: No test for adler32 (adler32-zlib) [ 875.090762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 875.163614] LNet: Added LNI 192.168.202.27@tcp1 [8/256/0/180] [ 875.877161] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 878.101218] LNet: 40734:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 878.108868] LNetError: 40734:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 878.116600] LNet: Removed LNI 192.168.202.27@tcp1 [ 878.132333] LNet: Removed LNI 10.1.2.3@tcp2 [ 878.628381] Key type .llcrypt unregistered [ 878.632323] Key type ._llcrypt unregistered [ 891.262892] Key type ._llcrypt registered [ 891.265203] Key type .llcrypt registered [ 892.429347] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 23:31:00 (1782531060) [ 892.859930] LNet: 41537:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 892.859930] mask usage: [+|-] ... [ 893.030141] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 893.041817] alg: No test for adler32 (adler32-zlib) [ 893.971146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 894.035546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 894.084176] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 894.996596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 895.026360] LNet: 41703:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 895.032722] LNetError: 41703:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 895.048270] LNet: Removed LNI 192.168.202.27@tcp [ 895.065602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 895.190453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 895.227291] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 903.055858] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 23:31:11 (1782531071) [ 903.183783] LNet: 42287:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 903.201114] LNetError: 42287:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 903.229480] LNet: Removed LNI 192.168.202.27@tcp [ 903.807938] Key type .llcrypt unregistered [ 903.812841] Key type ._llcrypt unregistered [ 912.169359] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 912.510485] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 912.513335] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 912.520715] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 915.119755] Key type ._llcrypt registered [ 915.125547] Key type .llcrypt registered [ 915.200939] LNet: 42573:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 915.200939] mask usage: [+|-] ... [ 915.491243] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 915.532268] alg: No test for adler32 (adler32-zlib) [ 916.579434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 916.721664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 916.765672] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 917.931584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 917.956900] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 920.891594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 920.909880] LNet: 43039:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 920.914783] LNetError: 43039:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 920.919547] LNet: Removed LNI 10.1.2.3@tcp [ 920.933789] LNet: Removed LNI 192.168.202.27@tcp [ 920.951555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 921.011175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 921.053196] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 921.060878] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 922.127743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 922.161619] LNet: 43067:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 922.167333] LNetError: 43067:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 922.170190] LNetError: 43067:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 922.179994] LNet: Removed LNI 10.1.2.3@tcp [ 922.191537] LNet: Removed LNI 192.168.202.27@tcp [ 922.248808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 922.399907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 922.452880] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 922.458490] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 923.332501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 923.351712] LNet: Added LNI 192.168.202.27@tcp2 [8/256/0/180] [ 924.827575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 924.868784] LNet: 43229:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 924.881553] LNetError: 43229:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 924.886137] LNetError: 43229:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 924.910874] LNet: Removed LNI 10.1.2.3@tcp [ 924.912470] LNet: Removed LNI 192.168.202.27@tcp [ 924.936559] LNet: Removed LNI 192.168.202.27@tcp2 [ 924.954732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 925.066895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 925.118710] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 925.127812] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 925.137777] LNet: Added LNI 192.168.202.27@tcp2 [8/256/0/180] [ 934.424449] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 23:31:42 (1782531102) [ 934.893189] LNet: 43827:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 934.912199] LNetError: 43827:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 934.921268] LNetError: 43827:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 934.935536] LNet: Removed LNI 10.1.2.3@tcp [ 934.947871] LNet: Removed LNI 192.168.202.27@tcp [ 934.963781] LNet: Removed LNI 192.168.202.27@tcp2 [ 935.618831] Key type .llcrypt unregistered [ 935.625749] Key type ._llcrypt unregistered [ 942.545576] Key type ._llcrypt registered [ 942.548498] Key type .llcrypt registered [ 942.581648] LNet: 44052:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 942.581648] mask usage: [+|-] ... [ 942.692725] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 942.701927] alg: No test for adler32 (adler32-zlib) [ 943.667458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 943.741979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 943.769159] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 944.692819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 944.711202] LNet: 44221:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 944.715677] LNetError: 44221:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 944.728483] LNet: Removed LNI 192.168.202.27@tcp [ 944.772956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 944.839860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 944.889258] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 945.422128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 945.451576] LNet: 44251:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 945.457511] LNetError: 44251:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 945.473801] LNet: Removed LNI 192.168.202.27@tcp [ 945.496369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 945.562460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 945.586294] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 946.220977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 946.284649] LNet: Removed LNI 192.168.202.27@tcp [ 946.317558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 946.408768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 946.434795] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 946.812752] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 946.971995] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 946.976400] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 946.979661] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 949.437575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 949.455957] LNet: 44356:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 949.460911] LNet: 44356:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 949.466538] LNetError: 44356:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 949.474630] LNetError: 44356:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 949.487760] LNet: Removed LNI 192.168.202.27@tcp [ 949.511787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 949.566510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 949.606942] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 957.463709] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 23:32:04 (1782531124) [ 957.683309] LNet: 44948:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 957.696370] LNetError: 44948:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 957.726581] LNet: Removed LNI 192.168.202.27@tcp [ 958.457067] Key type .llcrypt unregistered [ 958.463694] Key type ._llcrypt unregistered [ 965.594662] Key type ._llcrypt registered [ 965.596743] Key type .llcrypt registered [ 965.631932] LNet: 45180:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 965.631932] mask usage: [+|-] ... [ 965.776483] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 965.790635] alg: No test for adler32 (adler32-zlib) [ 966.696860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 966.745798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 966.767501] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 967.379222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 967.395796] LNet: 45353:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 967.400718] LNetError: 45353:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 967.410940] LNet: Removed LNI 192.168.202.27@tcp [ 967.438913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 967.499633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 967.525471] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 974.426195] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 23:32:22 (1782531142) [ 974.634058] LNet: 45940:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 974.645342] LNetError: 45940:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 974.675436] LNet: Removed LNI 192.168.202.27@tcp [ 975.242286] Key type .llcrypt unregistered [ 975.244114] Key type ._llcrypt unregistered [ 983.463348] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 983.703180] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 983.705658] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 983.731739] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 986.471318] Key type ._llcrypt registered [ 986.472792] Key type .llcrypt registered [ 986.523678] LNet: 46222:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 986.523678] mask usage: [+|-] ... [ 986.715250] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 986.748818] alg: No test for adler32 (adler32-zlib) [ 987.726451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 987.778850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 987.803948] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 988.870417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 988.886992] LNet: 46396:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 988.890221] LNetError: 46396:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 988.905439] LNet: Removed LNI 192.168.202.27@tcp [ 988.916304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 988.945656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 988.967117] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 997.128497] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 23:32:44 (1782531164) [ 997.442475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 997.481886] LNet: 46995:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 997.487961] LNetError: 46995:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 997.513661] LNet: Removed LNI 192.168.202.27@tcp [ 997.542545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 997.671406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 1004.166289] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 23:32:52 (1782531172) [ 1004.425944] LNet: 47578:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1005.093402] Key type .llcrypt unregistered [ 1005.095596] Key type ._llcrypt unregistered [ 1011.720907] Key type ._llcrypt registered [ 1011.725469] Key type .llcrypt registered [ 1011.772932] LNet: 47803:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1011.772932] mask usage: [+|-] ... [ 1011.869319] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1011.880383] alg: No test for adler32 (adler32-zlib) [ 1012.817418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1012.882248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1012.906290] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 1013.931447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1013.954560] LNet: 47975:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1013.958764] LNetError: 47975:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1013.970511] LNet: Removed LNI 192.168.202.27@tcp [ 1013.990457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1014.081499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 1014.104063] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 1020.579425] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 23:33:08 (1782531188) [ 1020.854087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1020.897860] LNet: 48569:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1020.903052] LNetError: 48569:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1020.926051] LNet: Removed LNI 192.168.202.27@tcp [ 1020.964887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1021.036173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 1028.278352] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 23:33:15 (1782531195) [ 1030.391822] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 1032.583109] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 23:33:20 (1782531200) [ 1034.727863] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 1037.161703] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 23:33:24 (1782531204) [ 1039.134327] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 1040.994578] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 23:33:28 (1782531208) [ 1043.002227] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1044.738408] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 23:33:32 (1782531212) [ 1046.234872] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1048.267070] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 23:33:36 (1782531216) [ 1048.470879] LNet: 50624:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1048.946587] Key type .llcrypt unregistered [ 1048.948432] Key type ._llcrypt unregistered [ 1055.349286] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1055.649981] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1055.663974] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1055.677713] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1058.192708] Key type ._llcrypt registered [ 1058.194995] Key type .llcrypt registered [ 1058.234919] LNet: 50901:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1058.234919] mask usage: [+|-] ... [ 1058.476909] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1058.490938] alg: No test for adler32 (adler32-zlib) [ 1059.444510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1059.587949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1059.640494] LNet: Added LNI 192.168.202.27@tcp [8/512/0/180] [ 1059.651564] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1059.655216] LNet: 50939:0:(api-ni.c:2914:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1059.788638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1059.809139] LNet: 50958:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1059.813905] LNetError: 50958:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1059.818596] LNet: Removed LNI 10.1.2.3@tcp [ 1059.822407] LNetError: 50958:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1059.832636] LNet: Removed LNI 192.168.202.27@tcp [ 1059.848122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1059.891312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1059.909500] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 1059.955095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1059.969602] LNet: 50982:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1059.973399] LNetError: 50982:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1059.985738] LNet: Removed LNI 192.168.202.27@tcp [ 1059.998818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1060.042830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1060.064752] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 1060.075931] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1060.079169] LNet: 50990:0:(api-ni.c:2914:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1060.139421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1060.154661] LNet: 51008:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1060.158884] LNetError: 51008:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1060.163550] LNet: Removed LNI 10.1.2.3@tcp [ 1060.165594] LNetError: 51008:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1060.176641] LNet: Removed LNI 192.168.202.27@tcp [ 1060.188712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1060.245221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1060.263447] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 1060.267724] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1060.270337] LNet: 51016:0:(api-ni.c:2914:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1060.447347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1060.461853] LNet: 51042:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1060.466434] LNetError: 51042:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1060.471196] LNet: Removed LNI 10.1.2.3@tcp [ 1060.472840] LNetError: 51042:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1060.479849] LNet: Removed LNI 192.168.202.27@tcp [ 1060.492493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1060.553604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1060.567856] LNet: Added LNI 192.168.202.27@tcp [8/256/0/182] [ 1060.571278] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1060.664200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1060.689430] LNet: 51070:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1060.694977] LNetError: 51070:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1060.701831] LNet: Removed LNI 10.1.2.3@tcp [ 1060.704051] LNetError: 51070:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1060.723225] LNet: Removed LNI 192.168.202.27@tcp [ 1060.751391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1060.838754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1060.855473] LNet: Added LNI 192.168.202.27@tcp [8/256/0/182] [ 1060.859156] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1060.861568] LNet: 51078:0:(api-ni.c:2914:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1060.917822] LNet: Removed LNI 10.1.2.3@tcp [ 1060.925758] LNet: Removed LNI 192.168.202.27@tcp [ 1061.514405] Key type .llcrypt unregistered [ 1061.521216] Key type ._llcrypt unregistered [ 1074.077292] Key type ._llcrypt registered [ 1074.080256] Key type .llcrypt registered [ 1075.630780] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 23:34:03 (1782531243) [ 1075.943085] LNet: 51893:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1075.943085] mask usage: [+|-] ... [ 1076.206640] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1076.231031] alg: No test for adler32 (adler32-zlib) [ 1077.180879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1077.213393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1077.232427] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 1077.985191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.202.27 [ 1078.026379] LNet: 52054:0:(config.c:1627:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1078.041765] LNet: Removed LNI 192.168.202.27@tcp [ 1084.491298] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 23:34:12 (1782531252) [ 1084.747844] LNet: 52624:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1084.754576] LNetError: 52624:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1084.793454] LNet: Removed LNI 192.168.202.27@tcp [ 1085.350345] Key type .llcrypt unregistered [ 1085.360930] Key type ._llcrypt unregistered [ 1091.921793] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1092.098962] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1094.965210] Key type ._llcrypt registered [ 1094.968761] Key type .llcrypt registered [ 1095.024580] LNet: 52906:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1095.024580] mask usage: [+|-] ... [ 1095.165057] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1095.190941] alg: No test for adler32 (adler32-zlib) [ 1096.430492] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1102.622241] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 23:34:30 (1782531270) [ 1102.897393] LNet: 53538:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1102.902568] LNetError: 53538:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1102.921260] LNet: Removed LNI 10.1.2.3@tcp [ 1103.431566] Key type .llcrypt unregistered [ 1103.433160] Key type ._llcrypt unregistered [ 1111.236933] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1111.620454] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1111.630401] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1111.635760] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1114.704472] Key type ._llcrypt registered [ 1114.708667] Key type .llcrypt registered [ 1114.761275] LNet: 53812:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1114.761275] mask usage: [+|-] ... [ 1115.004509] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1115.035945] alg: No test for adler32 (adler32-zlib) [ 1116.134942] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1123.858925] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 23:34:51 (1782531291) [ 1125.663264] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1127.600313] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 23:34:55 (1782531295) [ 1128.072171] LNet: 54724:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1128.081572] LNetError: 54724:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1128.098937] LNet: Removed LNI 10.1.2.3@tcp [ 1128.710405] Key type .llcrypt unregistered [ 1128.713111] Key type ._llcrypt unregistered [ 1136.766523] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1136.989584] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1137.000381] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1137.012375] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1139.660795] Key type ._llcrypt registered [ 1139.662702] Key type .llcrypt registered [ 1139.690570] LNet: 55013:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1139.690570] mask usage: [+|-] ... [ 1139.861680] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1139.885757] alg: No test for adler32 (adler32-zlib) [ 1140.850986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1140.941988] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1151.056327] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 23:35:18 (1782531318) [ 1151.671354] LNet: 55970:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1151.678275] LNetError: 55970:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1151.698225] LNet: Removed LNI 10.1.2.3@tcp [ 1152.451237] Key type .llcrypt unregistered [ 1152.454333] Key type ._llcrypt unregistered [ 1160.011309] Key type ._llcrypt registered [ 1160.014562] Key type .llcrypt registered [ 1160.074379] LNet: 56193:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1160.074379] mask usage: [+|-] ... [ 1160.252522] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1160.266622] alg: No test for adler32 (adler32-zlib) [ 1161.230213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1161.264492] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 1171.279193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.127@tcp [ 1172.212919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.27@tcp -d 192.168.202.127@tcp -m GET -e local_interrupt -r 1 [ 1172.273678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.27@tcp -d 192.168.202.27@tcp -m GET -e local_interrupt -r 1 [ 1172.326757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.127@tcp [ 1172.363534] LNet: There was an unexpected network error while writing to 192.168.202.127: rc = -22 [ 1172.579650] LNet: 1 local NIs in recovery (showing 1): 192.168.202.27@tcp [ 1172.635868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1172.659868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1172.691826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1172.729842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1172.757807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1172.808414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1172.851895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1172.895822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1172.920774] LNet: 56409:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1172.936536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1172.971627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1173.001852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1173.029925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1173.062603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1173.105945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1173.144737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1173.760916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.27@tcp -d 192.168.202.127@tcp -m GET -e local_dropped -r 1 [ 1173.800393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.27@tcp -d 192.168.202.27@tcp -m GET -e local_dropped -r 1 [ 1173.847402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.127@tcp [ 1173.870835] LNet: There was an unexpected network error while writing to 192.168.202.127: rc = -22 [ 1174.087928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1174.125953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1174.157990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1174.192923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1174.218726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1174.247764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1174.280790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1174.317825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1174.340537] LNet: 56518:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1174.360697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1174.396590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1174.423562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1174.446677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1174.479510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1174.522248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1174.553967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1175.094584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.27@tcp -d 192.168.202.127@tcp -m GET -e local_aborted -r 1 [ 1175.132287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.27@tcp -d 192.168.202.27@tcp -m GET -e local_aborted -r 1 [ 1175.179803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.127@tcp [ 1175.197649] LNet: There was an unexpected network error while writing to 192.168.202.127: rc = -22 [ 1175.463178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1175.485667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1175.514759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1175.535566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1175.559478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1175.605898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1175.632823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1175.656092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1175.673779] LNet: 56627:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1175.698938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1175.766251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1175.828921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1175.866790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1175.914090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1175.957301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1176.021547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1176.672445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.27@tcp -d 192.168.202.127@tcp -m GET -e local_no_route -r 1 [ 1176.712346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.27@tcp -d 192.168.202.27@tcp -m GET -e local_no_route -r 1 [ 1176.771861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.127@tcp [ 1177.052419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1177.081606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1177.133548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1177.217366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1177.300736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1177.387972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1177.445779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1177.470738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1177.502375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1177.587628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1177.643924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1177.690327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1177.701864] LNet: There was an unexpected network error while writing to 192.168.202.27: rc = -22 [ 1177.711074] LNet: Skipped 1 previous similar message [ 1177.744134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1177.778318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1177.813743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1178.367364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.27@tcp -d 192.168.202.127@tcp -m GET -e local_timeout -r 1 [ 1178.403384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.27@tcp -d 192.168.202.27@tcp -m GET -e local_timeout -r 1 [ 1178.436836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.127@tcp [ 1178.619969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1178.657789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1178.714780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1178.773489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1178.806396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1178.844851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1178.872577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1178.914170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1178.941453] LNet: 56846:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1178.952587] LNet: 56846:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1178.985363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1179.024740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1179.070412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1179.131530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1179.177215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1179.236787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1179.286113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1179.859804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.27@tcp -d 192.168.202.127@tcp -m GET -e local_error -r 1 [ 1179.905890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.27@tcp -d 192.168.202.27@tcp -m GET -e local_error -r 1 [ 1179.952621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.127@tcp [ 1180.196727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1180.227597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1180.249314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1180.284662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1180.321880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1180.356681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1180.391544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1180.441490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1180.487239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1180.518205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1180.545789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1180.587199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1180.631227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1180.660140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1180.702489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1180.998160] LNet: 56996:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1181.014700] LNetError: 56996:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1181.033889] LNet: Removed LNI 192.168.202.27@tcp [ 1181.396501] Key type .llcrypt unregistered [ 1181.399390] Key type ._llcrypt unregistered [ 1201.594960] Key type ._llcrypt registered [ 1201.596776] Key type .llcrypt registered [ 1202.753653] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 23:36:10 (1782531370) [ 1203.909322] Key type .llcrypt unregistered [ 1203.910862] Key type ._llcrypt unregistered [ 1211.042959] Key type ._llcrypt registered [ 1211.045091] Key type .llcrypt registered [ 1211.083677] LNet: 58035:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1211.083677] mask usage: [+|-] ... [ 1211.272397] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1211.320845] alg: No test for adler32 (adler32-zlib) [ 1212.293700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1212.324459] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 1220.434861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.127@tcp [ 1231.417616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1231.463390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1231.486023] LNet: Added LNI 192.168.202.27@tcp1 [8/256/0/180] [ 1232.657479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.27@tcp -d 192.168.202.127@tcp -m GET -e local_interrupt -r 1 [ 1232.691132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.27@tcp -d 192.168.202.127@tcp1 -m GET -e local_interrupt -r 1 [ 1232.721969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.27@tcp -d 192.168.202.27@tcp -m GET -e local_interrupt -r 1 [ 1232.749625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.27@tcp -d 192.168.202.27@tcp1 -m GET -e local_interrupt -r 1 [ 1232.783776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.27@tcp1 -d 192.168.202.127@tcp -m GET -e local_interrupt -r 1 [ 1232.815610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.27@tcp1 -d 192.168.202.127@tcp1 -m GET -e local_interrupt -r 1 [ 1232.843214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.27@tcp1 -d 192.168.202.27@tcp -m GET -e local_interrupt -r 1 [ 1232.874588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.27@tcp1 -d 192.168.202.27@tcp1 -m GET -e local_interrupt -r 1 [ 1232.906074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.127@tcp [ 1232.919731] LNet: There was an unexpected network error while writing to 192.168.202.127: rc = -22 [ 1232.924785] LNet: 1 local NIs in recovery (showing 1): 192.168.202.27@tcp [ 1233.108344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1233.122187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1233.144702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1233.180348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1233.202613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1233.225771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1233.260305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1233.284205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1233.301177] LNet: 58465:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1233.315566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1233.338957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1233.367160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1233.387847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1233.405912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1233.431978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1233.463244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1234.039781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.27@tcp -d 192.168.202.127@tcp -m GET -e local_dropped -r 1 [ 1234.095684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.27@tcp -d 192.168.202.127@tcp1 -m GET -e local_dropped -r 1 [ 1234.137863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.27@tcp -d 192.168.202.27@tcp -m GET -e local_dropped -r 1 [ 1234.184105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.27@tcp -d 192.168.202.27@tcp1 -m GET -e local_dropped -r 1 [ 1234.241361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.27@tcp1 -d 192.168.202.127@tcp -m GET -e local_dropped -r 1 [ 1234.279917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.27@tcp1 -d 192.168.202.127@tcp1 -m GET -e local_dropped -r 1 [ 1234.319528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.27@tcp1 -d 192.168.202.27@tcp -m GET -e local_dropped -r 1 [ 1234.362145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.27@tcp1 -d 192.168.202.27@tcp1 -m GET -e local_dropped -r 1 [ 1234.396720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.127@tcp [ 1234.423759] LNet: There was an unexpected network error while writing to 192.168.202.127: rc = -22 [ 1234.428670] LNet: Skipped 2 previous similar messages [ 1234.607398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1234.637896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1234.667085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1234.706086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1234.737182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1234.781480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1234.818611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1234.852529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1234.877845] LNet: 58587:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1234.895539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1234.925573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1234.954405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1234.982521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1235.004797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1235.039334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1235.093522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1235.653536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.27@tcp -d 192.168.202.127@tcp -m GET -e local_aborted -r 1 [ 1235.690548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.27@tcp -d 192.168.202.127@tcp1 -m GET -e local_aborted -r 1 [ 1235.723913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.27@tcp -d 192.168.202.27@tcp -m GET -e local_aborted -r 1 [ 1235.751486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.27@tcp -d 192.168.202.27@tcp1 -m GET -e local_aborted -r 1 [ 1235.784530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.27@tcp1 -d 192.168.202.127@tcp -m GET -e local_aborted -r 1 [ 1235.819367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.27@tcp1 -d 192.168.202.127@tcp1 -m GET -e local_aborted -r 1 [ 1235.849484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.27@tcp1 -d 192.168.202.27@tcp -m GET -e local_aborted -r 1 [ 1235.887668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.27@tcp1 -d 192.168.202.27@tcp1 -m GET -e local_aborted -r 1 [ 1235.916882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.127@tcp [ 1235.939728] LNet: There was an unexpected network error while writing to 192.168.202.127: rc = -22 [ 1235.942210] LNet: Skipped 2 previous similar messages [ 1236.190376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1236.211623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1236.240612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1236.270524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1236.309410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1236.335584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1236.371428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1236.398894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1236.416292] LNet: 58708:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1236.434759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1236.462195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1236.484754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1236.506268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1236.536696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1236.560725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1236.587511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1237.249386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.27@tcp -d 192.168.202.127@tcp -m GET -e local_no_route -r 1 [ 1237.303894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.27@tcp -d 192.168.202.127@tcp1 -m GET -e local_no_route -r 1 [ 1237.375237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.27@tcp -d 192.168.202.27@tcp -m GET -e local_no_route -r 1 [ 1237.438303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.27@tcp -d 192.168.202.27@tcp1 -m GET -e local_no_route -r 1 [ 1237.486349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.27@tcp1 -d 192.168.202.127@tcp -m GET -e local_no_route -r 1 [ 1237.523727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.27@tcp1 -d 192.168.202.127@tcp1 -m GET -e local_no_route -r 1 [ 1237.574773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.27@tcp1 -d 192.168.202.27@tcp -m GET -e local_no_route -r 1 [ 1237.642683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.27@tcp1 -d 192.168.202.27@tcp1 -m GET -e local_no_route -r 1 [ 1237.694163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.127@tcp [ 1237.980514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1238.044375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1238.101660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1238.145237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1238.177690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1238.210181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1238.240745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1238.295184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1238.342500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1238.368212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1238.394627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1238.445858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1238.481982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1238.538729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1238.577561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1239.297274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.27@tcp -d 192.168.202.127@tcp -m GET -e local_timeout -r 1 [ 1239.326718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.27@tcp -d 192.168.202.127@tcp1 -m GET -e local_timeout -r 1 [ 1239.368177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.27@tcp -d 192.168.202.27@tcp -m GET -e local_timeout -r 1 [ 1239.430869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.27@tcp -d 192.168.202.27@tcp1 -m GET -e local_timeout -r 1 [ 1239.476925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.27@tcp1 -d 192.168.202.127@tcp -m GET -e local_timeout -r 1 [ 1239.522436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.27@tcp1 -d 192.168.202.127@tcp1 -m GET -e local_timeout -r 1 [ 1239.556567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.27@tcp1 -d 192.168.202.27@tcp -m GET -e local_timeout -r 1 [ 1239.591473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.27@tcp1 -d 192.168.202.27@tcp1 -m GET -e local_timeout -r 1 [ 1239.633387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.127@tcp [ 1239.653362] LNet: There was an unexpected network error while writing to 192.168.202.127: rc = -22 [ 1239.660872] LNet: Skipped 5 previous similar messages [ 1239.869613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1239.892749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1239.917185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1239.946581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1239.974144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1239.998385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1240.019207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1240.040914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1240.058072] LNet: 58951:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1240.062616] LNet: 58951:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1240.075634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1240.105439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1240.127877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1240.152294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1240.176841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1240.201532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1240.226965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1240.772757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.27@tcp -d 192.168.202.127@tcp -m GET -e local_error -r 1 [ 1240.820737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.27@tcp -d 192.168.202.127@tcp1 -m GET -e local_error -r 1 [ 1240.881752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.27@tcp -d 192.168.202.27@tcp -m GET -e local_error -r 1 [ 1240.930353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.27@tcp -d 192.168.202.27@tcp1 -m GET -e local_error -r 1 [ 1240.982386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.27@tcp1 -d 192.168.202.127@tcp -m GET -e local_error -r 1 [ 1241.057923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.27@tcp1 -d 192.168.202.127@tcp1 -m GET -e local_error -r 1 [ 1241.102333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.27@tcp1 -d 192.168.202.27@tcp -m GET -e local_error -r 1 [ 1241.184027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.27@tcp1 -d 192.168.202.27@tcp1 -m GET -e local_error -r 1 [ 1241.241736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.127@tcp [ 1241.494492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1241.519737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1241.583268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1241.641565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1241.728873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1241.763493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1241.822047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1241.869647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1241.921877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1241.974416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1242.006202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1242.045921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1242.100786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1242.160459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1242.237852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1248.987846] LNet: 59148:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1248.992694] LNetError: 59148:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1250.017975] LNet: Removed LNI 192.168.202.27@tcp [ 1250.022132] LNetError: 59148:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1250.038971] LNet: Removed LNI 192.168.202.27@tcp1 [ 1250.576543] Key type .llcrypt unregistered [ 1250.578179] Key type ._llcrypt unregistered [ 1270.645982] Key type ._llcrypt registered [ 1270.647863] Key type .llcrypt registered [ 1272.869074] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 23:37:19 (1782531439) [ 1273.746047] Key type .llcrypt unregistered [ 1273.752338] Key type ._llcrypt unregistered [ 1281.402338] Key type ._llcrypt registered [ 1281.403830] Key type .llcrypt registered [ 1281.447593] LNet: 60186:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1281.447593] mask usage: [+|-] ... [ 1281.698167] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1281.721941] alg: No test for adler32 (adler32-zlib) [ 1282.592755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1282.620302] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 1291.492634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.127@tcp [ 1292.149340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.27@tcp -d 192.168.202.127@tcp -m GET -e remote_dropped -r 1 [ 1292.187988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.27@tcp -d 192.168.202.27@tcp -m GET -e remote_dropped -r 1 [ 1292.227194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.127@tcp [ 1292.250704] LNet: There was an unexpected network error while writing to 192.168.202.127: rc = -22 [ 1292.604761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1292.664448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1292.696816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1292.746673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1292.829185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1292.858891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1292.895622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1292.923722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1292.940240] LNet: 60402:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1292.965654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1293.001466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1293.047165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1293.089337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1293.128539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1293.171763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1293.215962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1293.882882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.27@tcp -d 192.168.202.127@tcp -m GET -e remote_error -r 1 [ 1293.924732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.27@tcp -d 192.168.202.27@tcp -m GET -e remote_error -r 1 [ 1293.963901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.127@tcp [ 1293.987159] LNet: There was an unexpected network error while writing to 192.168.202.127: rc = -22 [ 1294.185912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1294.228771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1294.257518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1294.281143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1294.307827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1294.337908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1294.366788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1294.392727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1294.409457] LNet: 60511:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1294.425312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1294.454550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1294.478222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1294.504899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1294.537593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1294.561960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1294.592302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1295.122919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.27@tcp -d 192.168.202.127@tcp -m GET -e remote_timeout -r 1 [ 1295.180175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.27@tcp -d 192.168.202.27@tcp -m GET -e remote_timeout -r 1 [ 1295.223829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.127@tcp [ 1295.241427] LNet: There was an unexpected network error while writing to 192.168.202.127: rc = -22 [ 1295.501213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1295.575646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1295.614638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1295.643965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1295.665054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1295.688983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1295.712508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1295.742713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1295.758828] LNet: 60621:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1295.776367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1295.808968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1295.839907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1295.881915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1295.916750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1295.952148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1295.988640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1296.246486] LNet: 60662:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1296.253861] LNetError: 60662:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1296.272645] LNet: Removed LNI 192.168.202.27@tcp [ 1296.644702] Key type .llcrypt unregistered [ 1296.647393] Key type ._llcrypt unregistered [ 1315.658370] Key type ._llcrypt registered [ 1315.660489] Key type .llcrypt registered [ 1316.878743] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 23:38:04 (1782531484) [ 1317.875354] Key type .llcrypt unregistered [ 1317.879794] Key type ._llcrypt unregistered [ 1324.367883] Key type ._llcrypt registered [ 1324.370356] Key type .llcrypt registered [ 1324.407474] LNet: 61700:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1324.407474] mask usage: [+|-] ... [ 1324.601536] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1324.610387] alg: No test for adler32 (adler32-zlib) [ 1325.568687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1325.596385] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 1333.828315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.127@tcp [ 1343.524762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1343.568502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1343.600552] LNet: Added LNI 192.168.202.27@tcp1 [8/256/0/180] [ 1345.493708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.27@tcp -d 192.168.202.127@tcp -m GET -e remote_dropped -r 1 [ 1345.531844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.27@tcp -d 192.168.202.127@tcp1 -m GET -e remote_dropped -r 1 [ 1345.569355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.27@tcp -d 192.168.202.27@tcp -m GET -e remote_dropped -r 1 [ 1345.610685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.27@tcp -d 192.168.202.27@tcp1 -m GET -e remote_dropped -r 1 [ 1345.665297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.27@tcp1 -d 192.168.202.127@tcp -m GET -e remote_dropped -r 1 [ 1345.711652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.27@tcp1 -d 192.168.202.127@tcp1 -m GET -e remote_dropped -r 1 [ 1345.750263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.27@tcp1 -d 192.168.202.27@tcp -m GET -e remote_dropped -r 1 [ 1345.791586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.27@tcp1 -d 192.168.202.27@tcp1 -m GET -e remote_dropped -r 1 [ 1345.841663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.127@tcp [ 1345.861568] LNet: There was an unexpected network error while writing to 192.168.202.127: rc = -22 [ 1345.866189] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.127@tcp [ 1346.159272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1346.198353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1346.238638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1346.280760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1346.336405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1346.388664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1346.425740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1346.475620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1346.496617] LNet: 62130:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1346.519357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1346.582530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1346.611136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1346.641481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1346.682338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1346.726821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1346.788688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1346.837951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1347.562520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.27@tcp -d 192.168.202.127@tcp -m GET -e remote_error -r 1 [ 1347.643950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.27@tcp -d 192.168.202.127@tcp1 -m GET -e remote_error -r 1 [ 1347.712508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.27@tcp -d 192.168.202.27@tcp -m GET -e remote_error -r 1 [ 1347.790482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.27@tcp -d 192.168.202.27@tcp1 -m GET -e remote_error -r 1 [ 1347.857722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.27@tcp1 -d 192.168.202.127@tcp -m GET -e remote_error -r 1 [ 1347.923183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.27@tcp1 -d 192.168.202.127@tcp1 -m GET -e remote_error -r 1 [ 1347.976508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.27@tcp1 -d 192.168.202.27@tcp -m GET -e remote_error -r 1 [ 1348.039819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.27@tcp1 -d 192.168.202.27@tcp1 -m GET -e remote_error -r 1 [ 1348.134476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.127@tcp [ 1348.187366] LNet: There was an unexpected network error while writing to 192.168.202.127: rc = -22 [ 1348.192569] LNet: Skipped 2 previous similar messages [ 1348.476590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1348.525511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1348.561866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1348.612556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1348.674301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1348.707374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1348.742863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1348.803277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1348.847995] LNet: 62253:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1348.877414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1348.915736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1348.956706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1348.991772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1349.032818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1349.075813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1349.119486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1349.157834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1349.697496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.27@tcp -d 192.168.202.127@tcp -m GET -e remote_timeout -r 1 [ 1349.762801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.27@tcp -d 192.168.202.127@tcp1 -m GET -e remote_timeout -r 1 [ 1349.840858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.27@tcp -d 192.168.202.27@tcp -m GET -e remote_timeout -r 1 [ 1349.892490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.27@tcp -d 192.168.202.27@tcp1 -m GET -e remote_timeout -r 1 [ 1349.964584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.27@tcp1 -d 192.168.202.127@tcp -m GET -e remote_timeout -r 1 [ 1350.010299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.27@tcp1 -d 192.168.202.127@tcp1 -m GET -e remote_timeout -r 1 [ 1350.088618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.27@tcp1 -d 192.168.202.27@tcp -m GET -e remote_timeout -r 1 [ 1350.129679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.27@tcp1 -d 192.168.202.27@tcp1 -m GET -e remote_timeout -r 1 [ 1350.168687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.127@tcp [ 1350.200825] LNet: There was an unexpected network error while writing to 192.168.202.127: rc = -22 [ 1350.436732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1350.464346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1350.506099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1350.553775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1350.588324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1350.636820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1350.686846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1350.727554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1350.746506] LNet: 62376:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1350.761664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1350.786512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1350.816725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1350.861569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1350.931995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1350.987527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1351.045411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1351.083661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1356.441134] LNet: 62455:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1356.449602] LNetError: 62455:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1356.459537] LNet: Removed LNI 192.168.202.27@tcp [ 1356.474893] LNet: Removed LNI 192.168.202.27@tcp1 [ 1356.950415] Key type .llcrypt unregistered [ 1356.953588] Key type ._llcrypt unregistered [ 1374.198989] Key type ._llcrypt registered [ 1374.203169] Key type .llcrypt registered [ 1375.347993] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 23:39:03 (1782531543) [ 1376.619168] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1378.618583] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 23:39:06 (1782531546) [ 1379.460379] Key type .llcrypt unregistered [ 1379.464058] Key type ._llcrypt unregistered [ 1386.461705] Key type ._llcrypt registered [ 1386.464730] Key type .llcrypt registered [ 1386.497869] LNet: 63784:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1386.497869] mask usage: [+|-] ... [ 1386.681177] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1386.706375] alg: No test for adler32 (adler32-zlib) [ 1387.681698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1387.720180] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 1394.532791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.127@tcp [ 1395.195777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.27@tcp -d 192.168.202.127@tcp -m GET -e network_timeout -r 1 [ 1395.247157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.27@tcp -d 192.168.202.27@tcp -m GET -e network_timeout -r 1 [ 1395.321464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.127@tcp [ 1395.364816] LNet: There was an unexpected network error while writing to 192.168.202.127: rc = -22 [ 1395.704215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1395.746755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1395.788174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1395.824206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1395.876516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1395.906739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1395.936136] LNet: 1 local NIs in recovery (showing 1): 192.168.202.27@tcp [ 1395.956361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1396.000743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1396.020973] LNet: 63999:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1396.042555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1396.077391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1396.113243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1396.148331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1396.217386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1396.255160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1396.540859] LNet: 64038:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1396.550760] LNetError: 64038:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1397.611808] LNet: Removed LNI 192.168.202.27@tcp [ 1398.085455] Key type .llcrypt unregistered [ 1398.088167] Key type ._llcrypt unregistered [ 1415.615754] Key type ._llcrypt registered [ 1415.619655] Key type .llcrypt registered [ 1415.670598] LNet: 64489:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1415.670598] mask usage: [+|-] ... [ 1415.874698] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1415.895113] alg: No test for adler32 (adler32-zlib) [ 1416.863311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1416.896506] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 1424.162853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.127@tcp [ 1432.558668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1432.599828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1432.625824] LNet: Added LNI 192.168.202.27@tcp1 [8/256/0/180] [ 1433.861809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.27@tcp -d 192.168.202.127@tcp -m GET -e network_timeout -r 1 [ 1433.898987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.27@tcp -d 192.168.202.127@tcp1 -m GET -e network_timeout -r 1 [ 1433.931862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.27@tcp -d 192.168.202.27@tcp -m GET -e network_timeout -r 1 [ 1433.968082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.27@tcp -d 192.168.202.27@tcp1 -m GET -e network_timeout -r 1 [ 1434.024074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.27@tcp1 -d 192.168.202.127@tcp -m GET -e network_timeout -r 1 [ 1434.049909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.27@tcp1 -d 192.168.202.127@tcp1 -m GET -e network_timeout -r 1 [ 1434.077305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.27@tcp1 -d 192.168.202.27@tcp -m GET -e network_timeout -r 1 [ 1434.105762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.27@tcp1 -d 192.168.202.27@tcp1 -m GET -e network_timeout -r 1 [ 1434.135471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.127@tcp [ 1434.152051] LNet: There was an unexpected network error while writing to 192.168.202.127: rc = -22 [ 1434.337411] LNet: 1 local NIs in recovery (showing 1): 192.168.202.27@tcp [ 1434.345061] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.127@tcp [ 1434.353912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1434.378499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1434.402440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1434.440580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1434.464296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1434.498986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1434.522455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1434.551271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1434.566824] LNet: 64919:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1434.584848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1434.611521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1434.638761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1434.668881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1434.691909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1434.719651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1438.584417] LNet: There was an unexpected network error while writing to 192.168.202.127: rc = -22 [ 1438.914825] LNet: 64993:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1438.926458] LNetError: 64993:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1439.968189] LNet: Removed LNI 192.168.202.27@tcp [ 1439.969965] LNetError: 64993:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1439.980736] LNet: Removed LNI 192.168.202.27@tcp1 [ 1440.325479] Key type .llcrypt unregistered [ 1440.328333] Key type ._llcrypt unregistered [ 1453.730472] Key type ._llcrypt registered [ 1453.734979] Key type .llcrypt registered [ 1454.665730] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 23:40:22 (1782531622) [ 1454.868525] LNet: 65795:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1454.868525] mask usage: [+|-] ... [ 1454.983461] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1454.994425] alg: No test for adler32 (adler32-zlib) [ 1455.887881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1455.940666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1455.965221] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 1456.674380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1456.690835] LNet: Added LNI 192.168.202.27@tcp1 [8/256/0/180] [ 1457.407194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.27@tcp [ 1457.475243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1457.533683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1457.561795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1457.591683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1457.618456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1457.651249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.27@tcp [ 1457.665543] LNetError: 66094:0:(api-ni.c:4400:lnet_ni_set_healthv()) manually adding local NI 192.168.202.27@tcp to recovery [ 1457.953408] LNet: 1 local NIs in recovery (showing 1): 192.168.202.27@tcp [ 1457.955087] LNet: There was an unexpected network error while writing to 192.168.202.27: rc = -22 [ 1460.002239] LNet: There was an unexpected network error while writing to 192.168.202.27: rc = -22 [ 1461.104768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1464.096448] LNet: There was an unexpected network error while writing to 192.168.202.27: rc = -22 [ 1464.555948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1464.573922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1464.619370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1464.636425] LNet: 66201:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1464.641087] LNetError: 66201:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1464.651621] LNet: Removed LNI 192.168.202.27@tcp [ 1464.662476] LNet: Removed LNI 192.168.202.27@tcp1 [ 1464.679147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1464.723537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1464.745735] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 1465.225483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1465.241313] LNet: Added LNI 192.168.202.27@tcp1 [8/256/0/180] [ 1465.883559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.27@tcp [ 1465.912967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1465.947619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1465.975803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1466.001555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1466.029652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1466.067670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1466.094333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.27@tcp [ 1466.113371] LNetError: 66473:0:(api-ni.c:4400:lnet_ni_set_healthv()) manually adding local NI 192.168.202.27@tcp to recovery [ 1466.785308] LNet: There was an unexpected network error while writing to 192.168.202.27: rc = -22 [ 1469.675595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1472.928823] LNet: There was an unexpected network error while writing to 192.168.202.27: rc = -22 [ 1472.935620] LNet: Skipped 1 previous similar message [ 1477.665427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1477.688841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1477.710283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1483.610417] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 23:40:51 (1782531651) [ 1483.790933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1483.808623] LNet: 67191:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1483.813448] LNetError: 67191:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1483.816636] LNetError: 67191:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1483.822080] LNet: Removed LNI 192.168.202.27@tcp [ 1483.831222] LNet: Removed LNI 192.168.202.27@tcp1 [ 1483.846591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1483.896082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1483.929452] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 1484.623636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1484.648420] LNet: Added LNI 192.168.202.27@tcp1 [8/256/0/180] [ 1485.270845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.27@tcp [ 1485.338252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1485.363848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1485.399179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1485.423838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1485.442769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1485.467827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.27@tcp --health 0 [ 1485.548521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1485.920172] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.27@tcp [ 1486.946274] LNet: There was an unexpected network error while writing to 192.168.202.27: rc = -22 [ 1486.952387] LNet: Skipped 1 previous similar message [ 1500.896894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1501.098162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1501.139779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1501.217504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.27@tcp --health 500 [ 1501.328511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1508.004114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1508.070234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1508.103128] LNet: 67683:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1508.110444] LNetError: 67683:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1508.116387] LNetError: 67683:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1508.130854] LNet: Removed LNI 192.168.202.27@tcp [ 1508.156195] LNet: Removed LNI 192.168.202.27@tcp1 [ 1508.171826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1508.224512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1508.248494] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 1508.932360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1508.962732] LNet: Added LNI 192.168.202.27@tcp1 [8/256/0/180] [ 1509.807350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.27@tcp [ 1509.837830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1509.889740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1509.916149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1509.942433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1509.970727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1509.999742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1510.028958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.27@tcp --health 0 [ 1511.266331] LNet: There was an unexpected network error while writing to 192.168.202.27: rc = -22 [ 1511.272211] LNet: Skipped 3 previous similar messages [ 1512.421872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1521.755347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1521.823666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1521.876427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1528.537749] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 23:41:36 (1782531696) [ 1529.144258] LNet: 68732:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1529.151437] LNetError: 68732:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1529.156475] LNetError: 68732:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1529.162199] LNet: Removed LNI 192.168.202.27@tcp [ 1529.171720] LNet: Removed LNI 192.168.202.27@tcp1 [ 1529.648494] Key type .llcrypt unregistered [ 1529.654077] Key type ._llcrypt unregistered [ 1535.949379] Key type ._llcrypt registered [ 1535.951945] Key type .llcrypt registered [ 1535.997928] LNet: 68956:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1535.997928] mask usage: [+|-] ... [ 1536.245972] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1536.275572] alg: No test for adler32 (adler32-zlib) [ 1537.295224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1537.330359] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 1544.754795] Lustre: DEBUG MARKER: Initial discovery [ 1544.793935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.127@tcp [ 1546.605765] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1546.820711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1546.849450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.127@tcp [ 1546.869626] LNet: There was an unexpected network error while writing to 192.168.202.127: rc = -22 [ 1546.885372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1546.910647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.202.127@tcp -r 1 -m GET -l 3 [ 1546.939624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.202.127@tcp -r 1 -m GET -e local_error [ 1546.974249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.202.127@tcp -r 1 -m PUT -l 6 [ 1547.617666] LNet: 1 local NIs in recovery (showing 1): 192.168.202.27@tcp [ 1547.928687] Lustre: DEBUG MARKER: Force oleg227-client.virtnet to discover 192.168.202.127@tcp (in background) [ 1548.175962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.127@tcp [ 1549.241279] Lustre: DEBUG MARKER: Force oleg227-server to discover 192.168.202.27@tcp [ 1551.267192] LNet: There was an unexpected network error while writing to 192.168.202.127: rc = -22 [ 1551.352247] Lustre: DEBUG MARKER: Wait for 69496 [ 1552.574570] Lustre: DEBUG MARKER: Finished wait on 69496 [ 1552.882703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1552.919929] LNet: There was an unexpected network error while writing to 192.168.202.127: rc = -22 [ 1552.937020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1553.184625] LNet: 69945:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1553.193073] LNetError: 69945:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1553.206549] LNet: Removed LNI 192.168.202.27@tcp [ 1553.487444] Key type .llcrypt unregistered [ 1553.489740] Key type ._llcrypt unregistered [ 1569.680604] Key type ._llcrypt registered [ 1569.682909] Key type .llcrypt registered [ 1570.750525] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 23:42:18 (1782531738) [ 1571.192627] Key type .llcrypt unregistered [ 1571.194727] Key type ._llcrypt unregistered [ 1577.584320] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1577.753625] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1577.757606] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1577.761174] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1580.201793] Key type ._llcrypt registered [ 1580.204602] Key type .llcrypt registered [ 1580.237985] LNet: 70988:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1580.237985] mask usage: [+|-] ... [ 1580.336502] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1580.344271] alg: No test for adler32 (adler32-zlib) [ 1581.247681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1581.286719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1581.305885] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 1581.875677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1581.891114] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1588.636474] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 23:42:36 (1782531756) [ 1588.786531] LNet: 72015:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1588.795725] LNetError: 72015:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1588.801336] LNet: Removed LNI 10.1.2.3@tcp [ 1588.814171] LNet: Removed LNI 192.168.202.27@tcp [ 1589.315318] Key type .llcrypt unregistered [ 1589.319627] Key type ._llcrypt unregistered [ 1595.888636] Key type ._llcrypt registered [ 1595.893394] Key type .llcrypt registered [ 1595.946797] LNet: 72271:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1595.946797] mask usage: [+|-] ... [ 1596.135800] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1596.158247] alg: No test for adler32 (adler32-zlib) [ 1597.167311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1597.222183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1597.258243] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 1598.094721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1598.114604] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1598.905334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.202.27@tcp [ 1599.968297] LNetError: Unexpected error -22 connecting to 192.168.202.27@tcp at host 192.168.202.27:988 [ 1604.321914] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 23:42:52 (1782531772) [ 1604.749859] LNet: 73155:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1604.763148] LNetError: 73155:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1604.770127] LNet: Removed LNI 10.1.2.3@tcp [ 1604.787139] LNet: Removed LNI 192.168.202.27@tcp [ 1605.437583] Key type .llcrypt unregistered [ 1605.440849] Key type ._llcrypt unregistered [ 1610.669830] Key type ._llcrypt registered [ 1610.671864] Key type .llcrypt registered [ 1610.696290] LNet: 73386:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1610.696290] mask usage: [+|-] ... [ 1610.818610] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1610.839121] alg: No test for adler32 (adler32-zlib) [ 1611.726887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1611.776696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1611.803906] LNet: Added LNI 192.168.202.27@tcp1 [8/256/0/180] [ 1612.420211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1612.438529] LNet: Added LNI 192.168.202.27@tcp2 [8/256/0/180] [ 1613.142581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.202.27@tcp1 --nid 192.168.202.27@tcp2 [ 1616.062896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.27@tcp1 192.168.202.27@tcp2 [ 1616.083502] LNetError: 73990:0:(lib-move.c:2007:lnet_handle_find_routed_path()) Unable to send message from 192.168.202.27@tcp1 to 192.168.202.27@tcp2 - Route table may be misconfigured [ 1616.101918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.27@tcp2 192.168.202.27@tcp1 [ 1622.779877] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 23:43:10 (1782531790) [ 1622.872742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1622.914161] LNet: 74561:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1622.925796] LNetError: 74561:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1623.969045] LNet: Removed LNI 192.168.202.27@tcp1 [ 1623.974115] LNetError: 74561:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1623.988077] LNet: Removed LNI 192.168.202.27@tcp2 [ 1624.021705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1624.080897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1624.103748] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 1624.754717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1624.793656] LNet: Added LNI 192.168.202.27@tcp1 [8/256/0/180] [ 1625.919455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.27@tcp [ 1625.975037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.27@tcp 192.168.202.27@tcp [ 1626.035444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.27@tcp1 192.168.202.27@tcp1 [ 1626.072211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.27@tcp -d 192.168.202.27@tcp -e network_timeout [ 1626.143785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.27@tcp -d 192.168.202.27@tcp1 -e network_timeout [ 1626.181484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.27@tcp1 -d 192.168.202.27@tcp -e network_timeout [ 1626.220727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.27@tcp1 -d 192.168.202.27@tcp1 -e network_timeout [ 1626.288773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.27@tcp [ 1626.309259] LNet: There was an unexpected network error while writing to 192.168.202.27: rc = -22 [ 1626.332508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1626.481246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1626.610992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1627.105249] LNet: 1 local NIs in recovery (showing 1): 192.168.202.27@tcp [ 1632.883414] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 23:43:21 (1782531801) [ 1633.166845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1633.192737] LNet: 75430:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1633.198688] LNetError: 75430:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1633.206137] LNet: Removed LNI 192.168.202.27@tcp [ 1634.281442] LNet: Removed LNI 192.168.202.27@tcp1 [ 1634.340961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1634.406859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1634.680556] LNet: 75468:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1635.047460] Key type .llcrypt unregistered [ 1635.050408] Key type ._llcrypt unregistered [ 1646.854360] Key type ._llcrypt registered [ 1646.856845] Key type .llcrypt registered [ 1648.397333] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 23:43:36 (1782531816) [ 1648.732823] LNet: 76239:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1648.732823] mask usage: [+|-] ... [ 1648.917161] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1648.929189] alg: No test for adler32 (adler32-zlib) [ 1649.844969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1651.470228] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1653.260776] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 23:43:41 (1782531821) [ 1653.382563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1653.417914] LNet: 76560:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1653.473267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1653.577954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1653.611992] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 1654.260873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1654.280958] LNet: Added LNI 192.168.202.27@tcp1 [8/256/0/180] [ 1655.077976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.27@tcp [ 1655.118540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.27@tcp1 [ 1655.142852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.27@tcp1 [ 1660.863807] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 23:43:49 (1782531829) [ 1662.526852] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg227-server [ 1663.822352] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 23:43:52 (1782531832) [ 1665.059955] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg227-server [ 1666.593597] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 23:43:54 (1782531834) [ 1668.179823] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg227-server [ 1669.983671] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 23:43:58 (1782531838) [ 1671.333599] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg227-server [ 1672.595771] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 23:44:00 (1782531840) [ 1674.006672] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg227-server [ 1675.503539] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 23:44:03 (1782531843) [ 1677.040380] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg227-server [ 1678.412628] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 23:44:06 (1782531846) [ 1679.816629] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg227-server [ 1681.378266] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 23:44:09 (1782531849) [ 1682.748498] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg227-server [ 1683.754947] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 23:44:12 (1782531852) [ 1685.223995] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg227-server [ 1686.422418] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 23:44:14 (1782531854) [ 1686.704170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1686.721432] LNet: 80442:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1686.726504] LNetError: 80442:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1686.730952] LNet: Removed LNI 192.168.202.27@tcp [ 1687.784443] LNet: Removed LNI 192.168.202.27@tcp1 [ 1687.840709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1687.912533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1687.948554] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 1688.434497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1688.515000] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.27@tcp [ 1688.665537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1688.683453] LNet: 80590:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1688.688540] LNetError: 80590:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1688.693409] LNetError: 80590:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1688.703212] LNet: Removed LNI 192.168.202.27@tcp [ 1688.714883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1688.756871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1688.776423] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 1689.394727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.27@tcp [ 1689.512722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1690.598616] LNet: Removed LNI 192.168.202.27@tcp [ 1690.627453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1690.673506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1690.706984] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 1691.209964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1691.261559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.27@tcp [ 1691.343911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1691.352327] LNet: 80882:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1691.354799] LNet: 80882:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1691.357718] LNetError: 80882:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1691.360728] LNetError: 80882:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1691.374389] LNet: Removed LNI 192.168.202.27@tcp [ 1691.387651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1691.433735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1691.452240] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 1692.097440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.27@tcp [ 1692.250133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1693.287661] LNet: Removed LNI 192.168.202.27@tcp [ 1693.305132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1693.382643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1693.406691] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 1694.008853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1694.074704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.27@tcp [ 1694.197867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1694.213915] LNetError: 81175:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1694.217595] LNetError: 81175:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1695.272539] LNet: Removed LNI 192.168.202.27@tcp [ 1695.295661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1695.340028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1695.374621] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 1695.983705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.27@tcp [ 1696.180251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1696.195996] LNet: 81320:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1696.200333] LNet: 81320:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1697.259816] LNet: Removed LNI 192.168.202.27@tcp [ 1697.295992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1697.390636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1697.409488] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 1698.038094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1698.136897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.27@tcp [ 1698.272201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1698.290645] LNetError: 81467:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1698.298383] LNetError: 81467:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1698.311599] LNet: Removed LNI 192.168.202.27@tcp [ 1698.325199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1698.389510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1698.415537] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 1699.126696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.27@tcp [ 1699.320929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1700.391899] LNet: Removed LNI 192.168.202.27@tcp [ 1700.414978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1700.476059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1700.502127] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 1701.095619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1701.165257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.27@tcp [ 1701.359440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1702.444437] LNet: Removed LNI 192.168.202.27@tcp [ 1702.480213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1702.541965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1702.567818] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 1703.209594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.27@tcp [ 1703.441970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1704.491286] LNet: Removed LNI 192.168.202.27@tcp [ 1704.527643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1704.622462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1704.642102] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 1705.407718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1705.510400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.27@tcp [ 1705.728531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1705.741992] LNet: 82052:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1705.746443] LNet: 82052:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1706.794409] LNet: Removed LNI 192.168.202.27@tcp [ 1706.828720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1706.904251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1706.964227] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 1707.798515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.27@tcp [ 1708.116129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1708.135143] LNetError: 82198:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1708.142658] LNetError: 82198:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1708.168129] LNet: Removed LNI 192.168.202.27@tcp [ 1708.190743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1708.286604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1708.312436] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 1709.036812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1709.114133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1709.159291] LNet: Removed LNI 192.168.202.27@tcp [ 1709.176274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1709.245215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1709.283226] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 1709.985285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1715.281245] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 23:44:43 (1782531883) [ 1715.563093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1715.623252] LNet: Removed LNI 192.168.202.27@tcp [ 1715.635881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1715.677857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1715.697979] LNet: Added LNI 192.168.202.27@tcp231 [8/256/0/180] [ 1716.608576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1716.645660] LNet: Removed LNI 192.168.202.27@tcp231 [ 1716.664291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1716.756778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1716.785621] LNet: Added LNI 192.168.202.27@tcp231 [8/256/0/180] [ 1716.794315] LNet: 83180:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1717.272223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1717.337818] LNet: Removed LNI 192.168.202.27@tcp231 [ 1717.358189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1717.390523] LNet: Added LNI 192.168.202.27@tcp231 [8/256/0/180] [ 1718.259502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1718.288020] LNet: Removed LNI 192.168.202.27@tcp231 [ 1718.315128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1718.395290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1718.442749] LNet: Added LNI 192.168.202.27@tcp231 [8/256/0/0] [ 1718.454389] LNet: 83332:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1718.738406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1718.759217] LNet: Removed LNI 192.168.202.27@tcp231 [ 1718.770329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1718.786735] LNet: Added LNI 192.168.202.27@tcp231 [8/256/0/0] [ 1719.882209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1719.908898] LNet: Removed LNI 192.168.202.27@tcp231 [ 1719.934302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1719.993595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1720.006494] LNet: Added LNI 192.168.202.27@tcp231 [8/256/0/60] [ 1720.009482] LNet: 83483:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1720.548714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1720.594904] LNet: Removed LNI 192.168.202.27@tcp231 [ 1720.613945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1720.657573] LNet: Added LNI 192.168.202.27@tcp231 [8/256/0/60] [ 1721.560239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1721.608695] LNet: Removed LNI 192.168.202.27@tcp231 [ 1721.627353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1721.675624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1721.695150] LNet: Added LNI 192.168.202.27@tcp231 [8/256/0/180] [ 1722.012720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1722.037642] LNet: Removed LNI 192.168.202.27@tcp231 [ 1722.049891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1722.067581] LNet: Added LNI 192.168.202.27@tcp231 [8/256/0/180] [ 1729.188448] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 23:44:57 (1782531897) [ 1729.272442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1729.291600] LNet: 84338:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1729.295576] LNet: 84338:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 1729.302338] LNetError: 84338:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1729.308490] LNetError: 84338:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1729.329337] LNet: Removed LNI 192.168.202.27@tcp231 [ 1729.353302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1729.451664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1729.483365] LNet: Added LNI 192.168.202.27@tcp232 [8/256/0/180] [ 1730.182417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1730.260908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1730.293482] LNet: Removed LNI 192.168.202.27@tcp232 [ 1730.311139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1730.331076] LNet: Added LNI 192.168.202.27@tcp232 [8/256/0/180] [ 1731.255157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1731.337937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1731.386864] LNet: Removed LNI 192.168.202.27@tcp232 [ 1731.413694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1731.465678] LNet: Added LNI 192.168.202.27@tcp232 [8/256/0/180] [ 1732.129528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1732.169794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1732.210230] LNet: Removed LNI 192.168.202.27@tcp232 [ 1732.235974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1732.251110] LNet: Added LNI 192.168.202.27@tcp232 [8/256/0/180] [ 1732.844386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1732.872911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1732.897879] LNet: Removed LNI 192.168.202.27@tcp232 [ 1732.909201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1737.586148] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 23:45:06 (1782531906) [ 1738.281318] Key type .llcrypt unregistered [ 1738.282630] Key type ._llcrypt unregistered [ 1742.934747] Key type ._llcrypt registered [ 1742.938912] Key type .llcrypt registered [ 1742.972472] LNet: 85701:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1742.972472] mask usage: [+|-] ... [ 1743.082242] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1743.096950] alg: No test for adler32 (adler32-zlib) [ 1743.971787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1743.988548] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 1750.305971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.127@tcp [ 1756.427387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1756.469619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1756.501659] LNet: Added LNI 192.168.202.27@tcp1 [8/256/0/180] [ 1757.619581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.27@tcp -d 192.168.202.127@tcp -m GET -e local_interrupt -r 2 [ 1757.638890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.27@tcp -d 192.168.202.127@tcp1 -m GET -e local_interrupt -r 2 [ 1757.666438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.27@tcp -d 192.168.202.27@tcp -m GET -e local_interrupt -r 2 [ 1757.688165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.27@tcp -d 192.168.202.27@tcp1 -m GET -e local_interrupt -r 2 [ 1757.706223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.27@tcp1 -d 192.168.202.127@tcp -m GET -e local_interrupt -r 2 [ 1757.726971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.27@tcp1 -d 192.168.202.127@tcp1 -m GET -e local_interrupt -r 2 [ 1757.752338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.27@tcp1 -d 192.168.202.27@tcp -m GET -e local_interrupt -r 2 [ 1757.777290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.27@tcp1 -d 192.168.202.27@tcp1 -m GET -e local_interrupt -r 2 [ 1757.818980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.27@tcp 192.168.202.127@tcp [ 1757.839432] LNet: There was an unexpected network error while writing to 192.168.202.127: rc = -22 [ 1757.843670] LNet: 1 local NIs in recovery (showing 1): 192.168.202.27@tcp [ 1758.042845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1758.081568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1758.117242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1758.138634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1758.168901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1758.194676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1758.217141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1758.238641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1758.251647] LNet: 86133:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1758.266600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1758.287774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1758.313602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1758.335929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1758.360537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1758.381164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1758.750705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.27@tcp -d 192.168.202.127@tcp -m GET -e local_dropped -r 2 [ 1758.779962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.27@tcp -d 192.168.202.127@tcp1 -m GET -e local_dropped -r 2 [ 1758.812287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.27@tcp -d 192.168.202.27@tcp -m GET -e local_dropped -r 2 [ 1758.833977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.27@tcp -d 192.168.202.27@tcp1 -m GET -e local_dropped -r 2 [ 1758.864065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.27@tcp1 -d 192.168.202.127@tcp -m GET -e local_dropped -r 2 [ 1758.897570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.27@tcp1 -d 192.168.202.127@tcp1 -m GET -e local_dropped -r 2 [ 1758.951485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.27@tcp1 -d 192.168.202.27@tcp -m GET -e local_dropped -r 2 [ 1758.983984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.27@tcp1 -d 192.168.202.27@tcp1 -m GET -e local_dropped -r 2 [ 1759.012024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.27@tcp 192.168.202.127@tcp [ 1759.022440] LNet: There was an unexpected network error while writing to 192.168.202.127: rc = -22 [ 1759.222326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1759.243487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1759.264955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1759.294702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1759.315642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1759.347540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1759.374699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1759.401991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1759.422422] LNet: 86253:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1759.436184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1759.465712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1759.491961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1759.520530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1759.543662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1759.566900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1759.969724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.27@tcp -d 192.168.202.127@tcp -m GET -e local_aborted -r 2 [ 1759.994767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.27@tcp -d 192.168.202.127@tcp1 -m GET -e local_aborted -r 2 [ 1760.022305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.27@tcp -d 192.168.202.27@tcp -m GET -e local_aborted -r 2 [ 1760.049727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.27@tcp -d 192.168.202.27@tcp1 -m GET -e local_aborted -r 2 [ 1760.089444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.27@tcp1 -d 192.168.202.127@tcp -m GET -e local_aborted -r 2 [ 1760.121985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.27@tcp1 -d 192.168.202.127@tcp1 -m GET -e local_aborted -r 2 [ 1760.163698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.27@tcp1 -d 192.168.202.27@tcp -m GET -e local_aborted -r 2 [ 1760.198400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.27@tcp1 -d 192.168.202.27@tcp1 -m GET -e local_aborted -r 2 [ 1760.239595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.27@tcp 192.168.202.127@tcp [ 1760.272225] LNet: There was an unexpected network error while writing to 192.168.202.127: rc = -22 [ 1760.517318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1760.534504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1760.561106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1760.587330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1760.617970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1760.656055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1760.692908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1760.733427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1760.751687] LNet: 86374:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1760.762480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1760.787055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1760.823383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1760.867938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1760.895690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1760.924683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1761.435465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.27@tcp -d 192.168.202.127@tcp -m GET -e local_no_route -r 2 [ 1761.464732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.27@tcp -d 192.168.202.127@tcp1 -m GET -e local_no_route -r 2 [ 1761.495091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.27@tcp -d 192.168.202.27@tcp -m GET -e local_no_route -r 2 [ 1761.526786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.27@tcp -d 192.168.202.27@tcp1 -m GET -e local_no_route -r 2 [ 1761.554948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.27@tcp1 -d 192.168.202.127@tcp -m GET -e local_no_route -r 2 [ 1761.587319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.27@tcp1 -d 192.168.202.127@tcp1 -m GET -e local_no_route -r 2 [ 1761.619991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.27@tcp1 -d 192.168.202.27@tcp -m GET -e local_no_route -r 2 [ 1761.655305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.27@tcp1 -d 192.168.202.27@tcp1 -m GET -e local_no_route -r 2 [ 1761.700605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.27@tcp 192.168.202.127@tcp [ 1761.958274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1762.011971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1762.066081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1762.096613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1762.126926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1762.156267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1762.187625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1762.211459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1762.237479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1762.264709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1762.294914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1762.352250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1762.379688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1762.405945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1762.838521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.27@tcp -d 192.168.202.127@tcp -m GET -e local_timeout -r 2 [ 1762.865256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.27@tcp -d 192.168.202.127@tcp1 -m GET -e local_timeout -r 2 [ 1762.896767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.27@tcp -d 192.168.202.27@tcp -m GET -e local_timeout -r 2 [ 1762.926268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.27@tcp -d 192.168.202.27@tcp1 -m GET -e local_timeout -r 2 [ 1762.952972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.27@tcp1 -d 192.168.202.127@tcp -m GET -e local_timeout -r 2 [ 1762.982718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.27@tcp1 -d 192.168.202.127@tcp1 -m GET -e local_timeout -r 2 [ 1763.013519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.27@tcp1 -d 192.168.202.27@tcp -m GET -e local_timeout -r 2 [ 1763.051327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.27@tcp1 -d 192.168.202.27@tcp1 -m GET -e local_timeout -r 2 [ 1763.087638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.27@tcp 192.168.202.127@tcp [ 1763.111335] LNet: There was an unexpected network error while writing to 192.168.202.127: rc = -22 [ 1763.115968] LNet: Skipped 1 previous similar message [ 1763.344828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1763.367653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1763.415852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1763.452297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1763.485741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1763.517787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1763.567870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1763.641849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1763.684555] LNet: 86614:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1763.692185] LNet: 86614:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1763.711588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1763.737854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1763.769844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1763.795446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1763.831492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1763.885376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1764.342775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.27@tcp -d 192.168.202.127@tcp -m GET -e remote_dropped -r 2 [ 1764.369523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.27@tcp -d 192.168.202.127@tcp1 -m GET -e remote_dropped -r 2 [ 1764.400807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.27@tcp -d 192.168.202.27@tcp -m GET -e remote_dropped -r 2 [ 1764.427540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.27@tcp -d 192.168.202.27@tcp1 -m GET -e remote_dropped -r 2 [ 1764.452332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.27@tcp1 -d 192.168.202.127@tcp -m GET -e remote_dropped -r 2 [ 1764.478360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.27@tcp1 -d 192.168.202.127@tcp1 -m GET -e remote_dropped -r 2 [ 1764.507825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.27@tcp1 -d 192.168.202.27@tcp -m GET -e remote_dropped -r 2 [ 1764.541496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.27@tcp1 -d 192.168.202.27@tcp1 -m GET -e remote_dropped -r 2 [ 1764.579361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.27@tcp 192.168.202.127@tcp [ 1764.592240] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.127@tcp [ 1764.806933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1764.841584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1764.870389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1764.900824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1764.940396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1764.959592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1764.980500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1765.001156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1765.022586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1765.045230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1765.068934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1765.093587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1765.130910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1765.158901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1769.946868] LNet: 86808:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1769.951696] LNetError: 86808:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1769.958934] LNet: Removed LNI 192.168.202.27@tcp [ 1769.972455] LNet: Removed LNI 192.168.202.27@tcp1 [ 1770.368458] Key type .llcrypt unregistered [ 1770.370675] Key type ._llcrypt unregistered [ 1785.584312] Key type ._llcrypt registered [ 1785.586386] Key type .llcrypt registered [ 1786.730364] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 23:45:54 (1782531954) [ 1787.041896] LNet: 87613:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1787.041896] mask usage: [+|-] ... [ 1787.182040] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1787.194581] alg: No test for adler32 (adler32-zlib) [ 1788.071346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1788.124480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1788.150552] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 1788.866831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1788.894583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.27@tcp [ 1788.909586] LNetError: 87786:0:(api-ni.c:4400:lnet_ni_set_healthv()) manually adding local NI 192.168.202.27@tcp to recovery [ 1788.974787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.202.27@tcp [ 1789.059748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1794.707653] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 23:46:02 (1782531962) [ 1794.778181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1794.807710] LNet: 88371:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1794.810943] LNetError: 88371:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1794.822661] LNet: Removed LNI 192.168.202.27@tcp [ 1794.848358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1794.896835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1794.924323] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 1795.585253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1795.605311] LNet: Added LNI 192.168.202.27@tcp1 [8/256/0/180] [ 1796.472882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.202.27@tcp [ 1796.498806] LNetError: 88641:0:(api-ni.c:4400:lnet_ni_set_healthv()) manually adding local NI 192.168.202.27@tcp to recovery [ 1796.896461] LNet: 1 local NIs in recovery (showing 1): 192.168.202.27@tcp [ 1796.901868] LNet: There was an unexpected network error while writing to 192.168.202.27: rc = -22 [ 1797.752108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.202.27@tcp [ 1804.033832] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 23:46:12 (1782531972) [ 1805.341733] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg227-server [ 1806.663562] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 23:46:15 (1782531975) [ 1807.989877] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg227-server [ 1809.693819] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 23:46:17 (1782531977) [ 1810.990346] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg227-server [ 1812.348676] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 23:46:20 (1782531980) [ 1812.633068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1812.657851] LNet: 90264:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1812.661831] LNetError: 90264:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1813.664186] LNet: Removed LNI 192.168.202.27@tcp [ 1813.665990] LNetError: 90264:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1813.685204] LNet: Removed LNI 192.168.202.27@tcp1 [ 1813.707711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1813.769129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1813.794783] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 1814.248620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1814.366553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1814.408431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1814.458838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1814.502831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1814.530114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1819.044136] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 23:46:27 (1782531987) [ 1819.320512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1819.352266] LNet: 90989:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1819.359541] LNetError: 90989:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1819.377924] LNet: Removed LNI 192.168.202.27@tcp [ 1819.403157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1819.447393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1819.466451] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 1824.832793] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 23:46:33 (1782531993) [ 1824.968884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1824.996126] LNet: 91692:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1825.003440] LNetError: 91692:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1825.014206] LNet: Removed LNI 192.168.202.27@tcp [ 1825.032227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1825.081285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1825.104576] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 1830.372898] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 23:46:38 (1782531998) [ 1831.419515] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1832.700427] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 23:46:41 (1782532001) [ 1833.110824] LNet: 92727:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1833.118967] LNetError: 92727:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1833.138652] LNet: Removed LNI 192.168.202.27@tcp [ 1833.635777] Key type .llcrypt unregistered [ 1833.637735] Key type ._llcrypt unregistered [ 1838.837555] Key type ._llcrypt registered [ 1838.841757] Key type .llcrypt registered [ 1838.890129] LNet: 92956:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1838.890129] mask usage: [+|-] ... [ 1839.054164] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1839.070689] alg: No test for adler32 (adler32-zlib) [ 1839.928658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1839.954442] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 1845.948650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.127@tcp [ 1849.815357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.202.127@tcp [ 1850.086271] LNet: 93143:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1850.096621] LNetError: 93143:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1850.114488] LNet: Removed LNI 192.168.202.27@tcp [ 1850.615390] Key type .llcrypt unregistered [ 1850.618615] Key type ._llcrypt unregistered [ 1859.526076] Key type ._llcrypt registered [ 1859.530782] Key type .llcrypt registered [ 1860.315509] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1861.370979] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1862.426299] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 23:47:10 (1782532030) [ 1863.772222] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1864.840773] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 23:47:13 (1782532033) [ 1866.409380] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg227-server [ 1867.810298] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 23:47:16 (1782532036) [ 1869.219381] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg227-server [ 1870.421088] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 23:47:18 (1782532038) [ 1871.108673] LNet: 95193:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1871.108673] mask usage: [+|-] ... [ 1871.440061] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1871.475398] alg: No test for adler32 (adler32-zlib) [ 1872.468855] Lustre: Lustre: Build Version: 2.17.54_99_gde77dd7 [ 1872.621255] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 1874.240826] Key type lgssc registered [ 1875.306479] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1889.594360] Key type lgssc unregistered [ 1889.786423] LNet: 96315:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1889.794790] LNetError: 96315:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1889.816559] LNet: Removed LNI 192.168.202.27@tcp [ 1890.350517] Key type .llcrypt unregistered [ 1890.354916] Key type ._llcrypt unregistered [ 1901.119139] Key type ._llcrypt registered [ 1901.121274] Key type .llcrypt registered [ 1902.074412] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 23:47:50 (1782532070) [ 1902.353918] LNet: 97111:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1902.353918] mask usage: [+|-] ... [ 1902.477869] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1902.492960] alg: No test for adler32 (adler32-zlib) [ 1903.374464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1903.426787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1903.463650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1903.505468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1903.557518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1908.004151] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 23:47:56 (1782532076) [ 1908.189193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1908.230878] LNet: 97728:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1908.262290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1908.310914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1908.342830] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 1909.171094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1909.192662] LNet: 97872:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1909.196736] LNetError: 97872:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1909.207297] LNet: Removed LNI 192.168.202.27@tcp [ 1909.223769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1909.299904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1909.330164] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 1909.900330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1909.917848] LNetError: 97900:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1909.934580] LNet: Removed LNI 192.168.202.27@tcp [ 1909.962829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1910.023484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1910.065114] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 1915.973987] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 23:48:03 (1782532083) [ 1916.096217] LNet: 98485:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1916.103299] LNet: 98485:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1916.117384] LNetError: 98485:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1916.153084] LNet: Removed LNI 192.168.202.27@tcp [ 1916.673574] Key type .llcrypt unregistered [ 1916.679944] Key type ._llcrypt unregistered [ 1921.656247] Key type ._llcrypt registered [ 1921.658726] Key type .llcrypt registered [ 1921.687687] LNet: 98723:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1921.687687] mask usage: [+|-] ... [ 1921.804764] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1921.819380] alg: No test for adler32 (adler32-zlib) [ 1922.708470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1922.767238] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 1922.769220] LNet: *** cfs_fail_loc=e003, val=100*** [ 1923.242892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.27@tcp [ 1923.271223] LNet: *** cfs_fail_loc=e003, val=100*** [ 1923.275376] LNet: Skipped 4 previous similar messages [ 1923.278726] Ping reply received with mapping: [ 1923.284476] found metadata reply NI[192.168.202.27@tcp] [ 1923.291817] 1b ca a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1923.295431] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1923.300942] 5a 5a 5a 5a 5a 5a 5a 5a [ 1923.433847] LNet: 98889:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1923.443591] LNetError: 98889:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1923.460970] LNet: Removed LNI 192.168.202.27@tcp [ 1923.897290] Key type .llcrypt unregistered [ 1923.901523] Key type ._llcrypt unregistered [ 1932.354553] Key type ._llcrypt registered [ 1932.356945] Key type .llcrypt registered [ 1933.216532] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 23:48:21 (1782532101) [ 1933.620393] Key type .llcrypt unregistered [ 1933.622410] Key type ._llcrypt unregistered [ 1938.769834] Key type ._llcrypt registered [ 1938.771897] Key type .llcrypt registered [ 1938.803596] LNet: 99883:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1938.803596] mask usage: [+|-] ... [ 1938.935419] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1938.953804] alg: No test for adler32 (adler32-zlib) [ 1940.006725] LNetError: 99934:0:(api-ni.c:2779:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1940.338504] Key type .llcrypt unregistered [ 1940.341173] Key type ._llcrypt unregistered [ 1944.289362] Key type ._llcrypt registered [ 1944.291987] Key type .llcrypt registered [ 1945.329832] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 23:48:33 (1782532113) [ 1945.690104] Key type .llcrypt unregistered [ 1945.692242] Key type ._llcrypt unregistered [ 1950.475332] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1950.477039] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1952.887523] Key type ._llcrypt registered [ 1952.889931] Key type .llcrypt registered [ 1952.923795] LNet: 100801:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1952.923795] mask usage: [+|-] ... [ 1953.079519] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1953.092327] alg: No test for adler32 (adler32-zlib) [ 1954.050256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1954.117942] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1961.401161] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 23:48:49 (1782532129) [ 1961.490189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1961.520661] LNet: 101707:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1961.524898] LNetError: 101707:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1961.533991] LNet: Removed LNI 10.1.2.3@tcp [ 1961.549976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1962.184791] Key type .llcrypt unregistered [ 1962.188722] Key type ._llcrypt unregistered [ 1967.112560] Key type ._llcrypt registered [ 1967.117724] Key type .llcrypt registered [ 1967.151947] LNet: 101944:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1967.151947] mask usage: [+|-] ... [ 1967.319246] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1967.330740] alg: No test for adler32 (adler32-zlib) [ 1968.244890] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 1968.330536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.27@tcp [ 1968.586518] LNet: 102001:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1968.597613] LNetError: 102001:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1968.619428] LNet: Removed LNI 192.168.202.27@tcp [ 1969.253450] Key type .llcrypt unregistered [ 1969.256099] Key type ._llcrypt unregistered [ 1978.137269] Key type ._llcrypt registered [ 1978.139540] Key type .llcrypt registered [ 1979.263734] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 23:49:07 (1782532147) [ 1979.664573] Key type .llcrypt unregistered [ 1979.667411] Key type ._llcrypt unregistered [ 1984.685444] Key type ._llcrypt registered [ 1984.687432] Key type .llcrypt registered [ 1984.729672] LNet: 102984:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1984.729672] mask usage: [+|-] ... [ 1984.904328] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1984.934046] alg: No test for adler32 (adler32-zlib) [ 1990.933343] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 23:49:19 (1782532159) [ 1991.148507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1991.184874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1991.247404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1991.280170] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 1991.885285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1991.916964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1991.943807] LNet: Removed LNI 192.168.202.27@tcp [ 1992.127497] LNet: 103744:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1992.381399] Key type .llcrypt unregistered [ 1992.382849] Key type ._llcrypt unregistered [ 2001.607892] Key type ._llcrypt registered [ 2001.610167] Key type .llcrypt registered [ 2002.531458] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 23:49:30 (1782532170) [ 2002.813442] LNet: 104508:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2002.813442] mask usage: [+|-] ... [ 2002.927937] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2002.948642] alg: No test for adler32 (adler32-zlib) [ 2003.905202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2003.986535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2004.016713] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 2004.502580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.27@tcp [ 2004.540952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.202.27@tcp [ 2004.557868] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2004.560969] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2004.564478] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2004, 0, 0 [ 2004.567888] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2004.569404] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2004.571864] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2004.575389] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2004, 0, 0 [ 2004.578328] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2004.581496] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2004.585223] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2004.588473] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2004, 0, 2054 [ 2004.592904] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2004.595177] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2004.598794] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2004.602581] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2004, 0, 2054 [ 2004.608101] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2009.386644] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 23:49:37 (1782532177) [ 2009.743571] LNet: 105283:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2009.748840] LNetError: 105283:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2009.764901] LNet: Removed LNI 192.168.202.27@tcp [ 2010.224526] Key type .llcrypt unregistered [ 2010.226532] Key type ._llcrypt unregistered [ 2015.335202] Key type ._llcrypt registered [ 2015.337244] Key type .llcrypt registered [ 2015.372913] LNet: 105506:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2015.372913] mask usage: [+|-] ... [ 2015.496980] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2015.510764] alg: No test for adler32 (adler32-zlib) [ 2016.380138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2016.403757] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 2022.474871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.127@tcp [ 2029.101992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2029.145837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2029.181206] LNet: Added LNI 192.168.202.27@tcp1 [8/256/0/180] [ 2029.970921] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2030.076152] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2030.080128] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2030.083303] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2032.394634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2032.413960] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2040.380106] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 23:50:08 (1782532208) [ 2040.536746] LNet: 106780:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2040.542798] LNetError: 106780:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2041.568208] LNet: Removed LNI 192.168.202.27@tcp [ 2041.574072] LNetError: 106780:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2041.588308] LNet: Removed LNI 192.168.202.27@tcp1 [ 2041.602865] LNet: Removed LNI 10.1.2.3@tcp99 [ 2042.134577] Key type .llcrypt unregistered [ 2042.136835] Key type ._llcrypt unregistered [ 2047.031431] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2047.033764] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2049.477526] Key type ._llcrypt registered [ 2049.478666] Key type .llcrypt registered [ 2049.495604] LNet: 107053:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2049.495604] mask usage: [+|-] ... [ 2049.632642] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2049.648338] alg: No test for adler32 (adler32-zlib) [ 2050.516303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2050.550247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2050.564687] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 2051.005939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2051.029581] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2052.541721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2052.583493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2052.619974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2052.639699] LNetError: 107533:0:(peer.c:2234:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2052.668488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2052.721990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.27@tcp [ 2053.098444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2053.158337] LNet: 107553:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2053.169703] LNetError: 107553:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2053.172745] LNet: Removed LNI 10.1.2.3@tcp [ 2054.182836] LNet: Removed LNI 192.168.202.27@tcp [ 2054.670501] Key type .llcrypt unregistered [ 2054.672095] Key type ._llcrypt unregistered [ 2061.813633] Key type ._llcrypt registered [ 2061.817413] Key type .llcrypt registered [ 2062.625838] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 23:50:31 (1782532231) [ 2062.890837] LNet: 108351:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2062.890837] mask usage: [+|-] ... [ 2063.012839] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2063.026247] alg: No test for adler32 (adler32-zlib) [ 2063.886806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2063.921975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2063.933426] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 2064.456837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.27@tcp [ 2064.500492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg227-client@tcp [ 2068.860108] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 23:50:37 (1782532237) [ 2068.960676] LNet: 109080:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2068.968047] LNetError: 109080:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2068.984199] LNet: Removed LNI 192.168.202.27@tcp [ 2069.445360] Key type .llcrypt unregistered [ 2069.448507] Key type ._llcrypt unregistered [ 2073.947333] Key type ._llcrypt registered [ 2073.949136] Key type .llcrypt registered [ 2074.070802] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2074.078405] alg: No test for adler32 (adler32-zlib) [ 2074.980444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2074.994954] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 2075.012646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2079.459234] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 23:50:47 (1782532247) [ 2080.421702] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2081.774686] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 23:50:50 (1782532250) [ 2082.847668] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2084.006421] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 23:50:52 (1782532252) [ 2084.134525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2084.168559] LNet: 110500:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2084.173374] LNetError: 110500:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2084.201692] LNet: Removed LNI 192.168.202.27@tcp [ 2084.249910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2084.292370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2084.316672] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 2084.908060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2084.956843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2089.715939] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 23:50:58 (1782532258) [ 2089.789507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2089.807220] LNet: 111206:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2089.812927] LNetError: 111206:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2089.837133] LNet: Removed LNI 192.168.202.27@tcp [ 2089.859122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2089.942989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2089.970281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2090.599676] Key type .llcrypt unregistered [ 2090.601421] Key type ._llcrypt unregistered [ 2099.105846] Key type ._llcrypt registered [ 2099.107991] Key type .llcrypt registered [ 2099.920628] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 23:51:08 (1782532268) [ 2100.139687] LNet: 112018:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2100.139687] mask usage: [+|-] ... [ 2100.236266] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2100.250070] alg: No test for adler32 (adler32-zlib) [ 2101.136534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2101.176286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2101.193465] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 2101.699590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2101.756696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.27@tcp [ 2106.495413] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 23:51:14 (1782532274) [ 2106.552168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2106.571203] LNet: 112759:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2106.580587] LNetError: 112759:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2106.596887] LNet: Removed LNI 192.168.202.27@tcp [ 2106.621260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2106.659771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2106.696455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2111.061373] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 23:51:19 (1782532279) [ 2111.207895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2111.241331] LNet: 113346:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2111.270237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2111.321194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2111.613494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2111.629874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2111.652650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2111.947622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2111.982146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2112.037816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2112.067455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2112.129534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2112.407670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2112.450192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2112.493155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2112.829260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2112.849425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2117.360810] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 23:51:25 (1782532285) [ 2117.501989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2117.524602] LNet: 113977:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2117.532653] LNet: 113977:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2117.589307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2117.640376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2117.663650] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 2118.296147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2118.316245] LNet: Added LNI 192.168.202.27@tcp2 [8/256/0/180] [ 2118.916459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2118.957417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.27@tcp [ 2119.161964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2119.186307] LNetError: 114259:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2120.224688] LNet: Removed LNI 192.168.202.27@tcp [ 2120.227617] LNetError: 114259:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2120.250068] LNet: Removed LNI 192.168.202.27@tcp2 [ 2120.280566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2120.344624] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 2120.351933] LNet: Added LNI 192.168.202.27@tcp2 [8/256/0/180] [ 2120.359537] LNet: 114265:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2120.729388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.27@tcp [ 2120.889713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2120.905748] LNet: 114302:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2120.909996] LNet: 114302:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2121.952541] LNet: Removed LNI 192.168.202.27@tcp [ 2121.968641] LNet: Removed LNI 192.168.202.27@tcp2 [ 2121.987514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2122.036891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2122.053888] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 2122.526771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2122.541781] LNet: Added LNI 192.168.202.27@tcp2 [8/256/0/180] [ 2123.094928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.27@tcp [ 2123.116686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2123.154653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.27@tcp [ 2123.295731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2123.310230] LNetError: 114587:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2123.316188] LNetError: 114587:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2123.321340] LNet: Removed LNI 192.168.202.27@tcp [ 2124.393666] LNet: Removed LNI 192.168.202.27@tcp2 [ 2124.411959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2124.483417] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 2124.488232] LNet: Added LNI 192.168.202.27@tcp2 [8/256/0/180] [ 2124.492775] LNet: 114593:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2124.833816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.27@tcp [ 2124.947699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2124.970555] LNet: 114630:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2124.977809] LNet: 114630:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2124.984891] LNet: Removed LNI 192.168.202.27@tcp [ 2126.057556] LNet: Removed LNI 192.168.202.27@tcp2 [ 2126.076544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2126.124747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2126.142099] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 2126.606361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2126.618706] LNet: Added LNI 192.168.202.27@tcp2 [8/256/0/180] [ 2127.151838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2127.197834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.27@tcp [ 2127.351376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2127.368463] LNetError: 114909:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2127.373497] LNetError: 114909:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2127.383105] LNet: Removed LNI 192.168.202.27@tcp [ 2127.392987] LNet: Removed LNI 192.168.202.27@tcp2 [ 2127.406874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2127.455162] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 2127.459844] LNet: Added LNI 192.168.202.27@tcp2 [8/256/0/180] [ 2127.464377] LNet: 114915:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2127.758627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.27@tcp [ 2127.898755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2128.928262] LNet: Removed LNI 192.168.202.27@tcp [ 2128.942739] LNet: Removed LNI 192.168.202.27@tcp2 [ 2128.999468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2129.032222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2129.046172] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 2129.489971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2129.502427] LNet: Added LNI 192.168.202.27@tcp2 [8/256/0/180] [ 2130.031108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.27@tcp [ 2130.080105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2130.145487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.27@tcp [ 2130.300854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2131.360225] LNet: Removed LNI 192.168.202.27@tcp [ 2132.394160] LNet: Removed LNI 192.168.202.27@tcp2 [ 2132.434418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2132.492591] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 2132.498372] LNet: Added LNI 192.168.202.27@tcp2 [8/256/0/180] [ 2132.501794] LNet: 115243:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2132.883961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.27@tcp [ 2133.013267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2133.029310] LNet: 115280:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2133.033885] LNet: 115280:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2133.037919] LNet: Removed LNI 192.168.202.27@tcp [ 2134.056226] LNet: Removed LNI 192.168.202.27@tcp2 [ 2134.077579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2134.125847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2134.145765] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 2134.662428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2134.678580] LNet: Added LNI 192.168.202.27@tcp2 [8/256/0/180] [ 2135.241360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2135.259659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2135.310685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.27@tcp [ 2135.515259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2135.525843] LNetError: 115576:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2135.528542] LNetError: 115576:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2135.533600] LNet: Removed LNI 192.168.202.27@tcp [ 2135.542577] LNet: Removed LNI 192.168.202.27@tcp2 [ 2135.557907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2135.608454] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 2135.613064] LNet: Added LNI 192.168.202.27@tcp2 [8/256/0/180] [ 2135.917676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.27@tcp [ 2136.140656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2137.184157] LNet: Removed LNI 192.168.202.27@tcp [ 2137.192626] LNet: Removed LNI 192.168.202.27@tcp2 [ 2137.208074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2137.255342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2137.286421] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 2137.781164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2137.792471] LNet: Added LNI 192.168.202.27@tcp2 [8/256/0/180] [ 2138.268766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.27@tcp [ 2138.291088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2138.309242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2138.335547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.27@tcp [ 2138.505061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2139.552179] LNet: Removed LNI 192.168.202.27@tcp [ 2140.583778] LNet: Removed LNI 192.168.202.27@tcp2 [ 2140.598609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2140.643886] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 2140.646776] LNet: Added LNI 192.168.202.27@tcp2 [8/256/0/180] [ 2140.650746] LNet: 115942:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2140.656829] LNet: 115942:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 2140.994718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.27@tcp [ 2145.828147] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2147.134925] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 23:51:55 (1782532315) [ 2147.210713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2147.229096] LNet: Removed LNI 192.168.202.27@tcp [ 2148.264501] LNet: Removed LNI 192.168.202.27@tcp2 [ 2148.287727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2148.536742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2148.554265] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 2149.083813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2149.102634] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2149.838755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2149.852399] LNet: 116974:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2149.856316] LNet: 116974:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2149.861614] LNet: Removed LNI 10.1.2.3@tcp [ 2149.872701] LNet: Removed LNI 192.168.202.27@tcp [ 2149.884915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2149.939415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2149.958850] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 2149.963940] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2150.231936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2150.251138] LNet: Removed LNI 192.168.202.27@tcp [ 2150.259107] LNet: Removed LNI 10.1.2.3@tcp [ 2154.551568] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 23:52:03 (1782532323) [ 2154.776663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2154.827128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2154.950668] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2155.081890] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2155.095304] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2155.097985] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2157.385707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2157.408482] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 2157.909228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2157.923433] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2159.339027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.27@tcp [ 2159.389321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2159.458574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.202.27@tcp -m PUT -e local_timeout -r 1 [ 2159.500265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2159.562731] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2159.564897] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2159.738499] LNetError: 117642:0:(lib-move.c:4714:lnet_parse()) 192.168.202.27@tcp, src 192.168.202.27@tcp: Dropping PUT (error -108 looking up sender) [ 2160.803854] LNetError: 118102:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2160.814863] LNetError: 118102:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 9 previous similar messages [ 2160.822241] LNet: Removed LNI 10.1.2.3@tcp [ 2161.894138] LNet: Removed LNI 192.168.202.27@tcp [ 2162.161438] Key type .llcrypt unregistered [ 2162.162890] Key type ._llcrypt unregistered [ 2169.603202] Key type ._llcrypt registered [ 2169.604659] Key type .llcrypt registered [ 2170.487896] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 23:52:18 (1782532338) [ 2170.735689] LNet: 118873:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2170.735689] mask usage: [+|-] ... [ 2170.846591] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2170.854225] alg: No test for adler32 (adler32-zlib) [ 2171.751861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2172.062932] LNet: 118954:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2172.478482] Key type .llcrypt unregistered [ 2172.487340] Key type ._llcrypt unregistered [ 2176.694190] Key type ._llcrypt registered [ 2176.696396] Key type .llcrypt registered [ 2176.727867] LNet: 119183:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2176.727867] mask usage: [+|-] ... [ 2176.810394] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2176.824384] alg: No test for adler32 (adler32-zlib) [ 2177.690850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2177.707509] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 2183.600688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.127@tcp [ 2183.765810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2184.236623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.127@tcp [ 2184.411848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2184.429660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2184.456729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2184.481711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2184.507306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2184.539373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2184.569885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2184.593623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2184.609387] LNet: 119403:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2184.622097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2184.650815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2184.676674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2184.706337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2184.744231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2184.781314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2185.269245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2185.547657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2185.581322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2185.606688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2185.629684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2185.650318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2185.685485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2185.716516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2185.743529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2185.764606] LNet: 119520:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2185.786960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2185.814168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2185.846974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2185.874975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2185.889125] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.127@tcp [ 2185.909953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2185.936732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2186.387983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.127@tcp [ 2186.634049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2186.660652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2186.690355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2186.708290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2186.725398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2186.744696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2186.763929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2186.791550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2186.806682] LNet: 119637:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2186.824326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2186.862139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2186.882337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2186.921639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2186.961155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2187.011871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2187.552538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2187.744787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2187.769379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2187.792179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2187.815638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2187.838267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2187.865787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2187.898146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2187.931986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2187.963643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2187.988310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2188.014646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2188.043243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2188.072561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2188.102359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2188.436466] LNet: 119801:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2188.447169] LNetError: 119801:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2188.462746] LNet: Removed LNI 192.168.202.27@tcp [ 2188.808684] Key type .llcrypt unregistered [ 2188.812703] Key type ._llcrypt unregistered [ 2200.375726] Key type ._llcrypt registered [ 2200.377550] Key type .llcrypt registered [ 2201.255064] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 23:52:49 (1782532369) [ 2201.459079] LNet: 120605:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2201.459079] mask usage: [+|-] ... [ 2201.582509] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2201.595277] alg: No test for adler32 (adler32-zlib) [ 2202.449536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2202.715734] LNet: 120687:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2203.076549] Key type .llcrypt unregistered [ 2203.081642] Key type ._llcrypt unregistered [ 2207.335418] Key type ._llcrypt registered [ 2207.337402] Key type .llcrypt registered [ 2207.373735] LNet: 120916:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2207.373735] mask usage: [+|-] ... [ 2207.537933] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2207.568843] alg: No test for adler32 (adler32-zlib) [ 2208.415168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2208.428878] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 2214.133792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.127@tcp [ 2220.006182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2220.040238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2220.066817] LNet: Added LNI 192.168.202.27@tcp1 [8/256/0/180] [ 2220.691070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2221.117422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.127@tcp [ 2221.256744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2221.273740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2221.297789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2221.316933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2221.338603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2221.362460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2221.395629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2221.428753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2221.439695] LNet: 121337:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2221.457220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2221.477188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2221.497923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2221.523665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2221.549487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2221.568111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2221.999477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2222.161358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2222.181783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2222.200871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2222.223996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2222.244563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2222.267096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2222.286581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2222.303786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2222.316178] LNet: 121455:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2222.325324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2222.353486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2222.382473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2222.403565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2222.422492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2222.440553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2222.829568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.127@tcp [ 2222.998214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2223.024220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2223.046067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2223.070348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2223.088932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2223.109109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2223.139418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2223.171859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2223.197931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2223.222772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2223.236858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2223.257613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2223.283739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2223.305524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2223.705387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2223.779162] LNet: 2 peer NIs in recovery (showing 2): 192.168.202.127@tcp, 192.168.202.127@tcp1 [ 2223.878535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2223.904757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2223.932910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2223.953134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2223.975703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2224.005714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2224.027745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2224.052487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2224.067515] LNet: 121689:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2224.071772] LNet: 121689:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 2224.088347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2224.109958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2224.132403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2224.154769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2224.176938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2224.209320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2227.924690] LNet: 121769:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2227.929255] LNetError: 121769:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2227.932849] LNet: Removed LNI 192.168.202.27@tcp [ 2227.942338] LNet: Removed LNI 192.168.202.27@tcp1 [ 2228.156772] Key type .llcrypt unregistered [ 2228.158450] Key type ._llcrypt unregistered [ 2238.092366] Key type ._llcrypt registered [ 2238.096710] Key type .llcrypt registered [ 2238.827287] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 23:53:27 (1782532407) [ 2242.659434] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 23:53:31 (1782532411) [ 2246.250973] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 23:53:34 (1782532414) [ 2250.421072] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 23:53:38 (1782532418) [ 2254.827495] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 23:53:43 (1782532423) [ 2259.059846] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 23:53:47 (1782532427) [ 2263.000293] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 23:53:51 (1782532431) [ 2266.551490] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 23:53:55 (1782532435) [ 2270.570681] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 23:53:59 (1782532439) [ 2274.533050] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 23:54:03 (1782532443) [ 2278.253053] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 23:54:06 (1782532446) [ 2281.747485] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 23:54:10 (1782532450) [ 2285.486083] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 23:54:14 (1782532454) [ 2288.764475] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 23:54:17 (1782532457) [ 2292.356260] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 23:54:21 (1782532461) [ 2295.809427] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 23:54:24 (1782532464) [ 2299.271762] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 23:54:27 (1782532467) [ 2302.505642] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 23:54:31 (1782532471) [ 2302.612817] LNet: 132296:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2302.612817] mask usage: [+|-] ... [ 2302.679727] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2302.686211] alg: No test for adler32 (adler32-zlib) [ 2303.512037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2303.570415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2303.582655] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 2304.144110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2304.155733] LNet: 132468:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2304.159639] LNetError: 132468:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2304.173846] LNet: Removed LNI 192.168.202.27@tcp [ 2304.187143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2304.221442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2304.236262] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 2307.545390] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 23:54:36 (1782532476) [ 2310.989481] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 23:54:39 (1782532479) [ 2314.005966] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 23:54:42 (1782532482) [ 2317.622842] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 23:54:46 (1782532486) [ 2321.729557] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 23:54:50 (1782532490) [ 2326.412163] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 23:54:54 (1782532494) [ 2330.767628] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 23:54:59 (1782532499) [ 2334.833240] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 23:55:03 (1782532503) [ 2339.015457] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 23:55:07 (1782532507) [ 2342.965973] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 23:55:11 (1782532511) [ 2346.510264] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 23:55:15 (1782532515) [ 2350.174839] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 23:55:18 (1782532518) [ 2354.066969] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2087 sec ========== 23:55:22 (1782532522) [ 2354.230332] LNet: 139925:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2354.237338] LNetError: 139925:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2354.246927] LNet: Removed LNI 192.168.202.27@tcp [ 2354.696584] Key type .llcrypt unregistered [ 2354.698599] Key type ._llcrypt unregistered [ 2362.519786] Key type ._llcrypt registered [ 2362.525241] Key type .llcrypt registered [ 2362.664580] LNet: 140397:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2362.664580] mask usage: [+|-] ... [ 2362.792170] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2362.801641] alg: No test for adler32 (adler32-zlib) [ 2363.754600] Lustre: Lustre: Build Version: 2.17.54_99_gde77dd7 [ 2363.905694] LNet: Added LNI 192.168.202.27@tcp [8/256/0/180] [ 2365.544129] Key type lgssc registered [ 2366.157361] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2404.886239] Lustre: Mounted lustre-client [ 2404.988382] LNet: 141865:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2404.988382] mask usage: [+|-] ... [ 2408.216156] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2412.597444] Lustre: Unmounted lustre-client [ 2446.410769] Key type lgssc unregistered [ 2446.622517] LNet: 142848:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2446.632599] LNetError: 142848:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2446.653320] LNet: Removed LNI 192.168.202.27@tcp [ 2447.121239] Key type .llcrypt unregistered [ 2447.123575] Key type ._llcrypt unregistered