[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 438467827 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895288K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002382] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004015] kvm-guest: setup PV IPIs [ 0.007618] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.011017] LSM: Security Framework initializing [ 0.012054] Yama: becoming mindful. [ 0.013038] SELinux: Initializing. [ 0.014066] *** VALIDATE selinux *** [ 0.022724] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027349] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029080] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030104] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031114] *** VALIDATE tmpfs *** [ 0.033082] *** VALIDATE proc *** [ 0.035157] *** VALIDATE cgroup *** [ 0.035832] *** VALIDATE cgroup2 *** [ 0.036278] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037151] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038000] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038029] Spectre V2 : User space: Vulnerable [ 0.039010] Speculative Store Bypass: Vulnerable [ 0.042203] debug: unmapping init [mem 0xffffffffa1c59000-0xffffffffa1c60fff] [ 0.044241] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045767] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046024] ... version: 2 [ 0.047011] ... bit width: 48 [ 0.048012] ... generic registers: 4 [ 0.049012] ... value mask: 0000ffffffffffff [ 0.050014] ... max period: 00007fffffffffff [ 0.051012] ... fixed-purpose events: 3 [ 0.052008] ... event mask: 000000070000000f [ 0.053249] rcu: Hierarchical SRCU implementation. [ 0.055534] smp: Bringing up secondary CPUs ... [ 0.056633] x86: Booting SMP configuration: [ 0.057026] .... node #0, CPUs: #1 #2 #3 [ 0.060441] smp: Brought up 1 node, 4 CPUs [ 0.062012] smpboot: Max logical packages: 1 [ 0.063020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.130874] node 0 deferred pages initialised in 65ms [ 0.134091] devtmpfs: initialized [ 0.135287] x86/mm: Memory block size: 128MB [ 0.140512] gcov: version magic: 0x41383552 [ 0.143178] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.147121] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.150302] pinctrl core: initialized pinctrl subsystem [ 0.152158] [ 0.152693] ************************************************************* [ 0.154013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.156010] ** ** [ 0.158011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.160010] ** ** [ 0.161008] ** This means that this kernel is built to expose internal ** [ 0.163011] ** IOMMU data structures, which may compromise security on ** [ 0.165010] ** your system. ** [ 0.167009] ** ** [ 0.169011] ** If you see this message and you are not debugging the ** [ 0.171012] ** kernel, report this immediately to your vendor! ** [ 0.172012] ** ** [ 0.174012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.176013] ************************************************************* [ 0.178794] NET: Registered protocol family 16 [ 0.180354] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.182055] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.184047] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.187099] cpuidle: using governor menu [ 0.188716] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.191433] PCI: Using configuration type 1 for base access [ 0.193122] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.203094] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.204020] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.206037] cryptd: max_cpu_qlen set to 1000 [ 0.208243] ACPI: Added _OSI(Module Device) [ 0.209013] ACPI: Added _OSI(Processor Device) [ 0.211014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.213015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.218337] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.223383] ACPI: Interpreter enabled [ 0.225059] ACPI: PM: (supports S0 S3 S4 S5) [ 0.227016] ACPI: Using IOAPIC for interrupt routing [ 0.229129] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.232415] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.241819] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.243036] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.245018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.248118] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.253407] acpiphp: Slot [2] registered [ 0.255121] acpiphp: Slot [5] registered [ 0.256132] acpiphp: Slot [6] registered [ 0.258130] acpiphp: Slot [3] registered [ 0.259088] acpiphp: Slot [4] registered [ 0.261111] acpiphp: Slot [7] registered [ 0.262084] acpiphp: Slot [8] registered [ 0.263081] acpiphp: Slot [9] registered [ 0.265134] acpiphp: Slot [10] registered [ 0.267121] acpiphp: Slot [11] registered [ 0.268088] acpiphp: Slot [12] registered [ 0.269102] acpiphp: Slot [13] registered [ 0.271166] acpiphp: Slot [14] registered [ 0.272082] acpiphp: Slot [15] registered [ 0.273101] acpiphp: Slot [16] registered [ 0.275123] acpiphp: Slot [17] registered [ 0.277101] acpiphp: Slot [18] registered [ 0.278081] acpiphp: Slot [19] registered [ 0.279124] acpiphp: Slot [20] registered [ 0.280097] acpiphp: Slot [21] registered [ 0.281000] acpiphp: Slot [22] registered [ 0.281000] acpiphp: Slot [23] registered [ 0.283103] acpiphp: Slot [24] registered [ 0.284110] acpiphp: Slot [25] registered [ 0.286085] acpiphp: Slot [26] registered [ 0.287193] acpiphp: Slot [27] registered [ 0.289097] acpiphp: Slot [28] registered [ 0.290090] acpiphp: Slot [29] registered [ 0.292091] acpiphp: Slot [30] registered [ 0.293126] acpiphp: Slot [31] registered [ 0.295066] PCI host bridge to bus 0000:00 [ 0.296016] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.298024] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.300023] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.303026] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.305023] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.308027] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.310187] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.314092] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.316659] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.324652] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.329012] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.333020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.335015] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.337017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.340526] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.342896] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.345044] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.348671] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.353014] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.364016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.368019] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.374492] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.382014] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.386016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.400015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.409105] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.415015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.421015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.442016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.452261] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.454384] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.457497] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.458324] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.461263] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.464133] iommu: Default domain type: Passthrough [ 0.466282] SCSI subsystem initialized [ 0.468122] ACPI: bus type USB registered [ 0.469134] usbcore: registered new interface driver usbfs [ 0.471104] usbcore: registered new interface driver hub [ 0.473078] usbcore: registered new device driver usb [ 0.475209] pps_core: LinuxPPS API ver. 1 registered [ 0.477014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.478076] PTP clock support registered [ 0.479179] EDAC MC: Ver: 3.0.0 [ 0.482003] PCI: Using ACPI for IRQ routing [ 0.483886] NetLabel: Initializing [ 0.486013] NetLabel: domain hash size = 128 [ 0.487011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.489095] NetLabel: unlabeled traffic allowed by default [ 0.492288] vgaarb: loaded [ 0.493338] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.495020] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.502000] clocksource: Switched to clocksource kvm-clock [ 0.616195] VFS: Disk quotas dquot_6.6.0 [ 0.617882] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.621223] *** VALIDATE ramfs *** [ 0.622555] *** VALIDATE hugetlbfs *** [ 0.623933] pnp: PnP ACPI init [ 0.625842] pnp: PnP ACPI: found 6 devices [ 0.639868] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.642988] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.644467] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.646274] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.648085] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.650556] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.653021] NET: Registered protocol family 2 [ 0.655205] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.659441] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.662759] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.668167] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.671768] TCP: Hash tables configured (established 65536 bind 65536) [ 0.674658] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.677911] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.680916] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.684322] NET: Registered protocol family 1 [ 0.687570] RPC: Registered named UNIX socket transport module. [ 0.690032] RPC: Registered udp transport module. [ 0.691786] RPC: Registered tcp transport module. [ 0.693573] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.696030] NET: Registered protocol family 44 [ 0.697939] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.700683] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.702882] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.705120] PCI: CLS 0 bytes, default 64 [ 0.706901] Unpacking initramfs... [ 2.108473] debug: unmapping init [mem 0xffff967efcc64000-0xffff967efffcffff] [ 2.112398] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.114825] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.117920] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.618895] Initialise system trusted keyrings [ 2.620811] Key type blacklist registered [ 2.622957] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.632248] zbud: loaded [ 2.635454] *** VALIDATE nfs *** [ 2.636670] *** VALIDATE nfs4 *** [ 2.638075] pstore: using deflate compression [ 2.641842] Platform Keyring initialized [ 2.745841] NET: Registered protocol family 38 [ 2.747926] Key type asymmetric registered [ 2.749881] Asymmetric key parser 'x509' registered [ 2.751990] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.756823] io scheduler mq-deadline registered [ 2.758548] io scheduler kyber registered [ 2.759816] io scheduler bfq registered [ 2.761761] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.764648] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.767645] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.770698] ACPI: Power Button [PWRF] [ 2.775182] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.780744] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.791165] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.818833] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.847290] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.852671] Non-volatile memory driver v1.3 [ 2.854028] Linux agpgart interface v0.103 [ 2.887573] virtio_blk virtio1: [vda] 146144 512-byte logical blocks (74.8 MB/71.4 MiB) [ 2.891278] vda: detected capacity change from 0 to 74825728 [ 2.906212] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.909190] vdb: detected capacity change from 0 to 1073741824 [ 2.918337] libphy: Fixed MDIO Bus: probed [ 2.923238] usbcore: registered new interface driver usbserial_generic [ 2.928189] usbserial: USB Serial support registered for generic [ 2.930163] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.934292] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.935975] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.938358] mousedev: PS/2 mouse device common for all mice [ 2.940962] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.941237] rtc_cmos 00:05: RTC can wake from S4 [ 2.953821] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.955683] rtc_cmos 00:05: registered as rtc0 [ 2.959308] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.962463] intel_pstate: CPU model not supported [ 2.964253] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.966884] hid: raw HID events driver (C) Jiri Kosina [ 2.969491] usbcore: registered new interface driver usbhid [ 2.971646] usbhid: USB HID core driver [ 2.973418] drop_monitor: Initializing network drop monitor service [ 2.975744] Initializing XFRM netlink socket [ 2.978148] NET: Registered protocol family 10 [ 2.981225] Segment Routing with IPv6 [ 2.983847] NET: Registered protocol family 17 [ 2.986026] mpls_gso: MPLS GSO support [ 2.992541] RAS: Correctable Errors collector initialized. [ 2.994287] AVX version of gcm_enc/dec engaged. [ 2.995609] AES CTR mode by8 optimization enabled [ 3.066712] sched_clock: Marking stable (3066625439, 0)->(3994746859, -928121420) [ 3.070438] registered taskstats version 1 [ 3.073271] Loading compiled-in X.509 certificates [ 3.075467] zswap: loaded using pool lzo/zbud [ 3.099243] Key type big_key registered [ 3.111195] Key type encrypted registered [ 3.113766] ima: No TPM chip found, activating TPM-bypass! [ 3.116861] ima: Allocated hash algorithm: sha1 [ 3.119669] ima: No architecture policies found [ 3.122661] evm: Initialising EVM extended attributes: [ 3.125351] evm: security.selinux [ 3.126962] evm: security.ima [ 3.128546] evm: security.capability [ 3.130585] evm: HMAC attrs: 0x1 [ 3.133630] rtc_cmos 00:05: setting system clock to 2026-08-24 10:53:09 UTC (1787568789) [ 3.139807] debug: unmapping init [mem 0xffffffffa2c03000-0xffffffffa2dfffff] [ 3.142872] debug: unmapping init [mem 0xffffffffa1982000-0xffffffffa1c58fff] [ 3.151111] Write protecting the kernel read-only data: 28672k [ 3.154627] debug: unmapping init [mem 0xffffffffa0003000-0xffffffffa01fffff] [ 3.157228] debug: unmapping init [mem 0xffffffffa0914000-0xffffffffa09fffff] [ 3.186148] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.193966] systemd[1]: Detected virtualization kvm. [ 3.196034] systemd[1]: Detected architecture x86-64. [ 3.197890] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.222923] systemd[1]: No hostname configured. [ 3.224813] systemd[1]: Set hostname to . [ 3.226800] random: systemd: uninitialized urandom read (16 bytes read) [ 3.229160] systemd[1]: Initializing machine ID from random generator. [ 3.264047] random: ln: uninitialized urandom read (6 bytes read) [ 3.346929] random: systemd: uninitialized urandom read (16 bytes read) [ 3.350379] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.355758] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.364092] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... [ OK ] Reached target Swap. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local File Systems. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... [ OK ] Listening on udev Control Socket. Starting Journal Service... [ OK ] Reached target Timers. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.000517] device-mapper: uevent: version 1.0.3 [ 4.003112] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.797406] virtio_net virtio0 ens2: renamed from eth0 [ 4.861581] scsi host0: ata_piix [ 4.893610] scsi host1: ata_piix [ 4.895737] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.898712] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.228277] dracut-initqueue[596]: RTNETLINK answers: File exists [ 9.639169] random: crng init done [ 9.640983] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.092159] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Slices. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.412184] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.701303] SELinux: Disabled at runtime. [ 11.766151] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.775230] systemd[1]: Detected virtualization kvm. [ 11.777233] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.327653] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.331613] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.341520] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.349255] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.352558] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.359338] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.372722] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Switch Root. [ OK ] Listening on udev Kernel Socket. Mounting Huge Pages File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice User and Session Slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Mounting Kernel Debug File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target Slices. [ OK ] Listening on Process Core Dump Socket. Starting Apply Kernel Variables... [ OK ] Stopped target Initrd File Systems. Starting udev Coldplug all Devices... [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Initrd Root File System. Starting Remount Root and Kernel File Systems... Activating swap /dev/disk/by-label/SWAP... Mounting POSIX Message Queue File System... [ OK ] Created slice system-getty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target Paths. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ 12.533308] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.812135] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.151033] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.190507] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.251780] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.270607] EDAC sbridge: Ver: 1.1.2 [ 14.345628] Key type dns_resolver registered [ 14.654255] NFS: Registering the id_resolver key type [ 14.657209] Key type id_resolver registered [ 14.659145] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg452-client login: [ 41.701493] libcfs: loading out-of-tree module taints kernel. [ 41.754758] Key type ._llcrypt registered [ 41.756413] Key type .llcrypt registered [ 42.067189] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 42.074858] alg: No test for adler32 (adler32-zlib) [ 43.125228] Lustre: Lustre: Build Version: 2.17.57_80_ga77f2be [ 43.519929] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 45.159175] Key type lgssc registered [ 46.407518] Lustre: Echo OBD driver; http://www.lustre.org/ [ 188.788061] Lustre: Mounted lustre-client - version 2.17.57_80_ga77f2be [ 195.137975] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 210.313386] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing check_logdir /tmp/testlogs/ [ 214.495302] Lustre: lustre-OST0000-osc-ffff967f45d16000: disconnect after 22s idle [ 215.906908] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing yml_node [ 220.358526] Lustre: DEBUG MARKER: Client: 2.17.57.80 [ 223.198820] Lustre: DEBUG MARKER: MDS: 2.17.57.80 [ 226.411431] Lustre: DEBUG MARKER: OSS: 2.17.57.80 [ 228.068837] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Mon Aug 24 06:56:53 EDT 2026 [ 244.234235] Lustre: Unmounted lustre-client [ 267.700082] hrtimer: interrupt took 4090649 ns [ 269.916924] Key type lgssc unregistered [ 270.279352] LNet: 8443:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 270.287675] LNetError: 8443:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 270.332913] LNet: Removed LNI 192.168.204.52@tcp [ 271.291188] Key type .llcrypt unregistered [ 271.295521] Key type ._llcrypt unregistered [ 300.466846] Key type ._llcrypt registered [ 300.469256] Key type .llcrypt registered [ 300.674073] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 300.686599] alg: No test for adler32 (adler32-zlib) [ 301.654633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 301.765562] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 301.803283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 306.337667] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 306.672599] LNet: 9356:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 306.697550] LNetError: 9356:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 306.727929] LNet: Removed LNI 192.168.204.52@tcp [ 307.666570] Key type .llcrypt unregistered [ 307.672313] Key type ._llcrypt unregistered [ 315.425487] Key type ._llcrypt registered [ 315.427838] Key type .llcrypt registered [ 316.989690] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 06:58:21 (1787569101) [ 317.425795] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 317.438391] alg: No test for adler32 (adler32-zlib) [ 318.463435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 318.601045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 318.645751] LNet: 9772:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 318.668866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 318.749680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 318.777415] LNet: 9779:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 326.975460] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 06:58:31 (1787569111) [ 327.059240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 327.089703] LNet: 10350:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 327.125630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 327.244914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 335.901568] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 06:58:40 (1787569120) [ 336.147687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 336.190956] LNet: 10943:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 336.229781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 336.359787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 343.948471] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 06:58:48 (1787569128) [ 344.157093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 344.175948] LNet: 11530:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 344.192989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 344.268516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 352.291736] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 06:58:57 (1787569137) [ 352.431528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 352.468629] LNet: 12109:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 352.506685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 352.627720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 353.251567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 361.442295] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 06:59:05 (1787569145) [ 361.573935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 361.611404] LNet: 12699:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 361.658492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 361.785879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 362.620274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 363.617987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 372.290852] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 06:59:17 (1787569157) [ 372.667142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 372.755595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 372.867968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 381.695865] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 06:59:26 (1787569166) [ 381.912797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 381.971396] LNet: 13885:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 381.985420] LNet: 13885:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 382.018925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 382.112789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 382.149856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 382.952589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 383.021848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 383.754463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 383.828440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 384.609380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 384.655425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 385.372205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 385.422944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 386.089386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 386.130460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 386.748927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 386.781765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 387.044800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 387.087103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 387.724277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 387.793194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 388.429854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 388.479257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 388.914991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 388.985968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 389.493702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 389.543867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 390.061980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 390.125511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 398.961988] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 06:59:44 (1787569184) [ 399.191554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 399.231345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 399.291785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 399.330572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 406.718422] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 06:59:51 (1787569191) [ 407.034404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 407.085505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 407.172485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 407.247794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 414.071791] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 06:59:59 (1787569199) [ 414.216518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 414.259530] LNet: 15739:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 414.265703] LNet: 15739:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 414.306032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 414.373157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 414.431869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 421.259601] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 07:00:06 (1787569206) [ 421.334877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 421.397905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 421.522210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 421.580169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 430.009895] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 07:00:14 (1787569214) [ 430.155717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 430.217549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 430.322502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 430.363917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 438.087050] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 07:00:22 (1787569222) [ 438.513504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 438.576408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 438.651661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 438.703694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 447.595713] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 07:00:32 (1787569232) [ 447.766279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 447.882476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 448.036914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 449.019854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 459.049343] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 07:00:43 (1787569243) [ 459.448520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 459.527674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 459.825402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 460.145771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 467.895702] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 07:00:52 (1787569252) [ 467.999834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 468.078569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 468.248816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 468.529243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 476.684874] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 07:01:01 (1787569261) [ 477.044422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 477.082696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 477.470881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 477.660112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 484.653392] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 07:01:09 (1787569269) [ 484.938739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 484.959708] LNet: 20585:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 484.968767] LNet: 20585:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 484.991423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 485.064744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 485.112744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 492.559264] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 07:01:17 (1787569277) [ 492.690523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 492.769685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 492.893623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 492.984647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 500.168565] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 07:01:25 (1787569285) [ 500.598748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 500.642676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 500.709282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 508.686941] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 07:01:33 (1787569293) [ 508.960653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 508.993927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 509.088420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 516.461276] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 07:01:40 (1787569300) [ 516.541314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 516.624546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 516.771512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 517.404516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 525.696981] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 07:01:50 (1787569310) [ 526.108212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 526.167711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 526.256667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 526.291980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 533.958155] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 07:01:58 (1787569318) [ 534.217469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 534.253609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 534.323697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 534.373244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 542.357735] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 07:02:06 (1787569326) [ 542.465326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 542.513269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 542.578945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 542.665241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 551.004471] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 07:02:15 (1787569335) [ 551.490320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 551.560944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 551.752393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 551.839773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 562.516215] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 07:02:26 (1787569346) [ 562.650417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 562.735581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 562.849858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 562.942057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 562.969815] LNetError: 25901:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 563.036430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 563.090530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 569.965175] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 07:02:34 (1787569354) [ 570.232703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 570.309476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 570.425046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.GfDRlrCC2t [ 570.529716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.GfDRlrCC2t [ 570.601372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.GfDRlrCC2t [ 577.981480] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 07:02:42 (1787569362) [ 578.305323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 578.423163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 578.516925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 578.593717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 578.691433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 586.727632] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 07:02:51 (1787569371) [ 586.911554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 586.994610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 587.174502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 587.216321] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 588.787366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.52@tcp [ 590.283172] Lustre: Lustre: Build Version: 2.17.57_80_ga77f2be [ 592.239165] Key type lgssc registered [ 593.439462] Lustre: Echo OBD driver; http://www.lustre.org/ [ 596.883166] LNetError: 27662:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.204.52@tcp99 has no available nets [ 599.558480] Key type lgssc unregistered [ 599.931827] LNetError: 28772:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 599.944025] LNet: Removed LNI 192.168.204.52@tcp [ 600.614567] Key type .llcrypt unregistered [ 600.621521] Key type ._llcrypt unregistered [ 609.059287] Key type ._llcrypt registered [ 609.064525] Key type .llcrypt registered [ 610.728303] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 07:03:15 (1787569395) [ 611.401472] Key type .llcrypt unregistered [ 611.406702] Key type ._llcrypt unregistered [ 619.006536] Key type ._llcrypt registered [ 619.008689] Key type .llcrypt registered [ 619.231305] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 619.269314] alg: No test for adler32 (adler32-zlib) [ 620.166861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 620.224698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 620.242357] LNet: 29693:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 620.248143] LNet: 29693:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 620.253715] LNet: 29693:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 620.262971] LNet: Message forwarding will be enabled in 2s [ 623.327155] LNet: Message forwarding enabled [ 623.640965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 623.662836] LNet: Message forwarding will stop in 2s to 6s [ 627.426019] LNet: Message forwarding disabled [ 628.222273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 628.246878] LNet: 29791:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 628.261747] LNet: 29791:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 628.266899] LNet: 29791:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 628.280566] LNet: Message forwarding will be enabled in 2s [ 628.415581] LNet: 29804:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 629.109377] Key type .llcrypt unregistered [ 629.114218] Key type ._llcrypt unregistered [ 643.772380] Key type ._llcrypt registered [ 643.775340] Key type .llcrypt registered [ 644.875813] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 07:03:50 (1787569430) [ 645.284414] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 645.306872] alg: No test for adler32 (adler32-zlib) [ 646.248553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 646.306833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 652.112743] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 07:03:57 (1787569437) [ 652.209808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 652.276582] LNet: 31208:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 652.307222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 652.436402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 652.491973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 652.541243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 652.573703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 652.620512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 652.650177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 652.683820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 652.726946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 652.756514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 652.796700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 652.837899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 652.869545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 652.893583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 652.952302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 653.003717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 653.032156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 653.067319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 653.108495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 653.144144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 653.158909] LNetError: 31256:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 659.362625] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 07:04:04 (1787569444) [ 659.675826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 659.696682] LNet: 31827:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 659.714692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 659.765534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 667.119287] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 07:04:11 (1787569451) [ 669.003355] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg452-server [ 670.757182] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 07:04:15 (1787569455) [ 671.113914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 671.178940] LNet: 32759:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 671.218080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 671.290626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 671.347632] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 672.380613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.204.[53-57]@tcp [ 672.410994] LNet: 32892:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 672.459584] LNetError: 32763:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.54@tcp is being used as a gateway but routing feature is not turned on [ 672.468150] LNetError: 32763:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.54@tcp has gone from up to down [ 680.213412] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 07:04:25 (1787569465) [ 682.250408] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg452-server [ 684.895638] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 07:04:29 (1787569469) [ 685.005430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 685.046407] LNet: 33822:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 685.060298] LNetError: 33822:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 685.086902] LNet: Removed LNI 192.168.204.52@tcp [ 685.127992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 685.208887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 685.228704] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 685.886647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.204.[53-58/2]@tcp [ 685.926610] LNet: 33947:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 685.940459] LNet: 33947:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 685.971283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.204.[53-58/2]@tcp [ 685.991704] LNetError: 33833:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.55@tcp is being used as a gateway but routing feature is not turned on [ 686.015996] LNetError: 33833:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 8 previous similar messages [ 686.023079] LNetError: 33833:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp103 through 192.168.204.55@tcp has gone from up to down [ 686.029552] LNetError: 33833:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 693.640983] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 07:04:38 (1787569478) [ 693.915982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 693.953318] LNet: 34539:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 693.965171] LNetError: 34539:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 693.998173] LNet: Removed LNI 192.168.204.52@tcp [ 694.058904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 694.212301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 694.291298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 694.334587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 694.393766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 694.460514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 694.521972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 694.622926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 694.745192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 694.797975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 694.895418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 694.980907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 695.014790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 695.084301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 695.209370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 695.274228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 695.349582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 695.412826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 695.451782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 695.508497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 695.589484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 695.620261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 695.687729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 695.777238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 695.809190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 695.871387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 695.995354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 696.054862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 696.155213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 696.240546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 696.280891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 696.344435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 696.426515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 696.469670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 696.570939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 704.180692] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 07:04:48 (1787569488) [ 706.221466] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg452-server [ 708.342820] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 07:04:53 (1787569493) [ 710.177400] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg452-server [ 712.213767] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 07:04:57 (1787569497) [ 712.545691] LNet: 35941:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 712.550943] LNet: 35941:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 713.307355] Key type .llcrypt unregistered [ 713.311218] Key type ._llcrypt unregistered [ 721.819167] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 721.973302] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 721.980677] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 721.994176] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 724.553515] Key type ._llcrypt registered [ 724.557414] Key type .llcrypt registered [ 724.757339] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 724.772534] alg: No test for adler32 (adler32-zlib) [ 725.719933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 725.791767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 725.815527] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 726.566451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 726.604341] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 728.440925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 728.454201] LNet: Removed LNI 10.1.2.3@tcp [ 728.618902] LNet: 36666:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 728.629592] LNetError: 36666:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 728.660068] LNet: Removed LNI 192.168.204.52@tcp [ 729.610704] Key type .llcrypt unregistered [ 729.612470] Key type ._llcrypt unregistered [ 742.539815] Key type ._llcrypt registered [ 742.543235] Key type .llcrypt registered [ 743.930831] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 07:05:28 (1787569528) [ 744.422341] Key type .llcrypt unregistered [ 744.424130] Key type ._llcrypt unregistered [ 751.751301] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 751.921994] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 751.934023] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 751.944360] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 754.770707] Key type ._llcrypt registered [ 754.774050] Key type .llcrypt registered [ 755.072580] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 755.109281] alg: No test for adler32 (adler32-zlib) [ 756.074111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 756.129889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 756.153389] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 756.832959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 756.857994] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 759.326083] LNet: 38168:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 759.336760] LNetError: 38168:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 759.342032] LNet: Removed LNI 10.1.2.3@tcp [ 759.352788] LNet: Removed LNI 192.168.204.52@tcp [ 760.080816] Key type .llcrypt unregistered [ 760.083755] Key type ._llcrypt unregistered [ 774.134729] Key type ._llcrypt registered [ 774.139896] Key type .llcrypt registered [ 775.720153] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 07:06:00 (1787569560) [ 776.535644] Key type .llcrypt unregistered [ 776.542349] Key type ._llcrypt unregistered [ 784.281125] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 784.423873] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 784.429896] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 784.438120] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 787.208440] Key type ._llcrypt registered [ 787.210311] Key type .llcrypt registered [ 787.556360] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 787.581826] alg: No test for adler32 (adler32-zlib) [ 788.572341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 788.682528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 788.731704] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 791.353842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 791.389385] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 794.066850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 794.101503] LNet: Removed LNI 10.1.2.3@tcp [ 794.120770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 794.159625] LNet: Removed LNI 10.1.2.31@tcp [ 794.177630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 794.225425] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 795.282426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 795.305441] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 796.526397] LNet: 40099:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 796.542126] LNetError: 40099:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 796.548419] LNet: Removed LNI 10.1.2.31@tcp [ 796.560945] LNet: Removed LNI 10.1.2.3@tcp [ 797.332768] Key type .llcrypt unregistered [ 797.345535] Key type ._llcrypt unregistered [ 813.693592] Key type ._llcrypt registered [ 813.695264] Key type .llcrypt registered [ 815.534853] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 07:06:39 (1787569599) [ 816.075308] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 816.106471] alg: No test for adler32 (adler32-zlib) [ 817.156612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 817.248923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 817.296732] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 818.418338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 818.469170] LNet: 41058:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 818.478795] LNetError: 41058:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 818.504416] LNet: Removed LNI 192.168.204.52@tcp [ 818.528662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 818.568271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.204.52@tcp [ 818.606514] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 819.579811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.204.52@tcp [ 819.632591] LNet: Removed LNI 192.168.204.52@tcp [ 819.763786] LNet: 41193:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 820.445592] Key type .llcrypt unregistered [ 820.447939] Key type ._llcrypt unregistered [ 834.344322] Key type ._llcrypt registered [ 834.347874] Key type .llcrypt registered [ 836.010556] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 07:07:00 (1787569620) [ 838.584611] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg452-server [ 841.092333] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 07:07:05 (1787569625) [ 841.906297] Key type .llcrypt unregistered [ 841.909370] Key type ._llcrypt unregistered [ 850.404371] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 850.661808] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 850.669112] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 850.676691] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 853.621372] Key type ._llcrypt registered [ 853.635582] Key type .llcrypt registered [ 853.914835] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 853.935814] alg: No test for adler32 (adler32-zlib) [ 854.889782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 854.983624] LNet: Added LNI 192.168.204.52@tcp1 [8/256/0/180] [ 855.952888] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 859.256868] LNet: 43033:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 859.266226] LNetError: 43033:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 859.277797] LNet: Removed LNI 192.168.204.52@tcp1 [ 859.293539] LNet: Removed LNI 10.1.2.3@tcp2 [ 859.983661] Key type .llcrypt unregistered [ 859.986960] Key type ._llcrypt unregistered [ 872.529596] Key type ._llcrypt registered [ 872.538386] Key type .llcrypt registered [ 874.315847] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 07:07:38 (1787569658) [ 874.779829] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 874.811023] alg: No test for adler32 (adler32-zlib) [ 875.771520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 875.846461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 875.876707] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 876.887986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 876.919722] LNet: 44006:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 876.926675] LNetError: 44006:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 876.941661] LNet: Removed LNI 192.168.204.52@tcp [ 876.966907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 877.033336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 877.053406] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 885.966965] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 07:07:50 (1787569670) [ 886.575022] LNet: 44598:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 886.593865] LNetError: 44598:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 886.630081] LNet: Removed LNI 192.168.204.52@tcp [ 887.658486] Key type .llcrypt unregistered [ 887.660240] Key type ._llcrypt unregistered [ 895.662450] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 895.767742] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 895.772708] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 895.787497] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 898.607953] Key type ._llcrypt registered [ 898.611138] Key type .llcrypt registered [ 898.773770] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 898.782688] alg: No test for adler32 (adler32-zlib) [ 899.706349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 899.744792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 899.763634] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 900.592543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 900.610645] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 902.753884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 902.771592] LNet: 45345:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 902.778735] LNetError: 45345:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 902.784156] LNet: Removed LNI 10.1.2.3@tcp [ 902.793896] LNet: Removed LNI 192.168.204.52@tcp [ 902.811215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 902.882065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 902.910148] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 902.916432] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 903.244323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 903.282038] LNet: 45372:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 903.290533] LNetError: 45372:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 903.301309] LNetError: 45372:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 903.309725] LNet: Removed LNI 10.1.2.3@tcp [ 903.333238] LNet: Removed LNI 192.168.204.52@tcp [ 903.405781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 903.529374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 903.591967] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 903.601332] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 904.237553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 904.274609] LNet: Added LNI 192.168.204.52@tcp2 [8/256/0/180] [ 905.390594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 905.450360] LNet: 45535:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 905.462089] LNetError: 45535:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 905.471147] LNetError: 45535:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 905.481469] LNet: Removed LNI 10.1.2.3@tcp [ 905.487114] LNet: Removed LNI 192.168.204.52@tcp [ 905.511870] LNet: Removed LNI 192.168.204.52@tcp2 [ 905.545991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 905.641106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 905.682122] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 905.697035] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 905.709575] LNet: Added LNI 192.168.204.52@tcp2 [8/256/0/180] [ 913.770539] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 07:08:18 (1787569698) [ 914.174393] LNet: 46135:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 914.188174] LNetError: 46135:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 914.196713] LNetError: 46135:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 914.208934] LNet: Removed LNI 10.1.2.3@tcp [ 914.213275] LNet: Removed LNI 192.168.204.52@tcp [ 914.240063] LNet: Removed LNI 192.168.204.52@tcp2 [ 915.212106] Key type .llcrypt unregistered [ 915.218096] Key type ._llcrypt unregistered [ 923.030839] Key type ._llcrypt registered [ 923.035394] Key type .llcrypt registered [ 923.378287] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 923.391933] alg: No test for adler32 (adler32-zlib) [ 924.487890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 924.565341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 924.607123] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 925.840373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 925.872256] LNet: 46533:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 925.892679] LNetError: 46533:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 925.947252] LNet: Removed LNI 192.168.204.52@tcp [ 926.032781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 926.153997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 926.203938] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 927.001314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 927.031119] LNet: 46564:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 927.036337] LNetError: 46564:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 927.048839] LNet: Removed LNI 192.168.204.52@tcp [ 927.078697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 927.176748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 927.201988] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 927.994551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 928.040692] LNetError: 46594:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 928.060677] LNet: Removed LNI 192.168.204.52@tcp [ 928.095265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 928.193927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 928.263972] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 928.672207] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 928.827988] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 928.832679] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 928.844559] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 931.287970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 931.320512] LNet: 46666:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 931.325678] LNet: 46666:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 931.330943] LNetError: 46666:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 931.342384] LNet: Removed LNI 192.168.204.52@tcp [ 931.374364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 931.435507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 931.474409] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 938.513205] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 07:08:43 (1787569723) [ 939.018644] LNet: 47263:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 939.034867] LNetError: 47263:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 939.058524] LNet: Removed LNI 192.168.204.52@tcp [ 939.789281] Key type .llcrypt unregistered [ 939.793768] Key type ._llcrypt unregistered [ 947.011440] Key type ._llcrypt registered [ 947.021418] Key type .llcrypt registered [ 947.380364] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 947.403650] alg: No test for adler32 (adler32-zlib) [ 948.393181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 948.444693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 948.466836] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 949.315267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 949.336971] LNet: 47666:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 949.343261] LNetError: 47666:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 949.357421] LNet: Removed LNI 192.168.204.52@tcp [ 949.381564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 949.449701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 949.485081] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 957.119173] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 07:09:01 (1787569741) [ 957.389437] LNet: 48258:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 957.401385] LNetError: 48258:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 957.438858] LNet: Removed LNI 192.168.204.52@tcp [ 958.211984] Key type .llcrypt unregistered [ 958.216143] Key type ._llcrypt unregistered [ 965.105104] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 965.196739] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 965.208971] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 965.214252] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 967.966527] Key type ._llcrypt registered [ 967.968613] Key type .llcrypt registered [ 968.189426] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 968.200568] alg: No test for adler32 (adler32-zlib) [ 969.162514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 969.230829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 969.261761] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 970.253164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 970.280767] LNet: 48713:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 970.292658] LNetError: 48713:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 970.314266] LNet: Removed LNI 192.168.204.52@tcp [ 970.351088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 970.440148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 970.474668] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 977.872146] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 07:09:23 (1787569763) [ 978.338817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 978.362803] LNet: 49311:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 978.370865] LNetError: 49311:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 978.396412] LNet: Removed LNI 192.168.204.52@tcp [ 978.420430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 978.534470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 984.918804] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 07:09:29 (1787569769) [ 985.273166] LNet: 49893:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 985.858681] Key type .llcrypt unregistered [ 985.863078] Key type ._llcrypt unregistered [ 992.812071] Key type ._llcrypt registered [ 992.814380] Key type .llcrypt registered [ 992.960844] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 992.976989] alg: No test for adler32 (adler32-zlib) [ 993.907231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 993.979338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 994.031586] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 994.785934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 994.810853] LNet: 50294:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 994.815951] LNetError: 50294:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 994.834073] LNet: Removed LNI 192.168.204.52@tcp [ 994.860148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 994.931488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 994.953365] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1002.180764] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 07:09:46 (1787569786) [ 1002.314335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1002.338990] LNet: 50881:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1002.351343] LNetError: 50881:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1002.379439] LNet: Removed LNI 192.168.204.52@tcp [ 1002.406850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1002.520212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 1009.434280] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 07:09:54 (1787569794) [ 1011.354023] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 1013.056459] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 07:09:58 (1787569798) [ 1015.057931] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 1017.255764] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 07:10:01 (1787569801) [ 1018.967886] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 1021.000403] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 07:10:05 (1787569805) [ 1023.586514] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1025.411534] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 07:10:10 (1787569810) [ 1027.198535] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1029.164134] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 07:10:14 (1787569814) [ 1029.636392] LNet: 52942:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1030.330778] Key type .llcrypt unregistered [ 1030.333588] Key type ._llcrypt unregistered [ 1038.101741] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1038.242516] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1038.257226] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1038.271512] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1040.898591] Key type ._llcrypt registered [ 1040.905641] Key type .llcrypt registered [ 1041.085224] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1041.103210] alg: No test for adler32 (adler32-zlib) [ 1042.034477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1042.136299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1042.169691] LNet: Added LNI 192.168.204.52@tcp [8/512/0/180] [ 1042.178711] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1042.181665] LNet: 53267:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1042.276807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1042.292373] LNet: 53286:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1042.301805] LNetError: 53286:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1042.313709] LNet: Removed LNI 10.1.2.3@tcp [ 1042.320389] LNetError: 53286:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1042.335204] LNet: Removed LNI 192.168.204.52@tcp [ 1042.366768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1042.440350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1042.461415] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1042.512906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1042.543224] LNet: 53310:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1042.550751] LNetError: 53310:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1042.576075] LNet: Removed LNI 192.168.204.52@tcp [ 1042.605173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1042.671802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1042.700969] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1042.705330] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1042.708286] LNet: 53318:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1042.804702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1042.828647] LNet: 53336:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1042.834969] LNetError: 53336:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1042.844236] LNet: Removed LNI 10.1.2.3@tcp [ 1042.845916] LNetError: 53336:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1042.856927] LNet: Removed LNI 192.168.204.52@tcp [ 1042.877357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1042.946726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1042.973835] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1042.981591] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1042.985792] LNet: 53344:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1043.157228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1043.183578] LNet: 53370:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1043.204055] LNetError: 53370:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1043.210471] LNet: Removed LNI 10.1.2.3@tcp [ 1043.212866] LNetError: 53370:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1043.224889] LNet: Removed LNI 192.168.204.52@tcp [ 1043.244254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1043.364781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1043.393874] LNet: Added LNI 192.168.204.52@tcp [8/256/0/182] [ 1043.401987] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1043.498116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1043.525909] LNet: 53398:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1043.536137] LNetError: 53398:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1043.546170] LNet: Removed LNI 10.1.2.3@tcp [ 1043.550390] LNetError: 53398:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1043.576408] LNet: Removed LNI 192.168.204.52@tcp [ 1043.594881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1043.660633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1043.698580] LNet: Added LNI 192.168.204.52@tcp [8/256/0/182] [ 1043.712745] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1043.722682] LNet: 53406:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1043.862183] LNet: Removed LNI 10.1.2.3@tcp [ 1043.875523] LNet: Removed LNI 192.168.204.52@tcp [ 1044.489487] Key type .llcrypt unregistered [ 1044.499806] Key type ._llcrypt unregistered [ 1057.473922] Key type ._llcrypt registered [ 1057.478234] Key type .llcrypt registered [ 1058.972185] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 07:10:43 (1787569843) [ 1060.830654] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 1062.625618] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 07:10:47 (1787569847) [ 1063.240599] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1063.254935] alg: No test for adler32 (adler32-zlib) [ 1064.223848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1064.291368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1064.317610] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1065.194851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.204.52 [ 1065.228402] LNet: 54675:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1065.243507] LNet: Removed LNI 192.168.204.52@tcp [ 1071.794941] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 07:10:56 (1787569856) [ 1072.239175] LNet: 55245:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1072.247921] LNetError: 55245:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1072.272154] LNet: Removed LNI 192.168.204.52@tcp [ 1072.885699] Key type .llcrypt unregistered [ 1072.887901] Key type ._llcrypt unregistered [ 1080.195963] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1080.436070] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1083.255873] Key type ._llcrypt registered [ 1083.259553] Key type .llcrypt registered [ 1083.459213] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1083.491956] alg: No test for adler32 (adler32-zlib) [ 1084.943661] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1092.221498] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 07:11:17 (1787569877) [ 1092.472575] LNet: 56162:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1092.495742] LNetError: 56162:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1092.534556] LNet: Removed LNI 10.1.2.3@tcp [ 1093.231530] Key type .llcrypt unregistered [ 1093.234099] Key type ._llcrypt unregistered [ 1101.015369] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1101.261886] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1101.270114] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1101.276213] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1104.182756] Key type ._llcrypt registered [ 1104.185319] Key type .llcrypt registered [ 1104.473986] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1104.508711] alg: No test for adler32 (adler32-zlib) [ 1105.591627] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1113.168122] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 07:11:37 (1787569897) [ 1115.279528] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1117.569544] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 07:11:42 (1787569902) [ 1117.785972] LNet: 57348:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1117.796917] LNetError: 57348:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1117.827482] LNet: Removed LNI 10.1.2.3@tcp [ 1118.452923] Key type .llcrypt unregistered [ 1118.464542] Key type ._llcrypt unregistered [ 1126.832239] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1127.135596] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1127.141285] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1127.150373] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1128.877937] Key type ._llcrypt registered [ 1128.880956] Key type .llcrypt registered [ 1129.173985] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1129.199153] alg: No test for adler32 (adler32-zlib) [ 1130.144217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1130.218207] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1139.403558] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 07:12:04 (1787569924) [ 1140.195564] LNet: 58577:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1140.205845] LNetError: 58577:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1140.221859] LNet: Removed LNI 10.1.2.3@tcp [ 1140.952667] Key type .llcrypt unregistered [ 1140.955183] Key type ._llcrypt unregistered [ 1148.811309] Key type ._llcrypt registered [ 1148.815284] Key type .llcrypt registered [ 1149.056850] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1149.081535] alg: No test for adler32 (adler32-zlib) [ 1149.993725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1150.016821] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1159.734473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.152@tcp [ 1160.648849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -e local_interrupt -r 1 [ 1160.718702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -e local_interrupt -r 1 [ 1160.762742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.152@tcp [ 1160.803811] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 1161.072439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1161.119895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1161.185876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1161.226544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1161.262141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1161.329783] LNet: 1 local NIs in recovery (showing 1): 192.168.204.52@tcp [ 1161.335886] LNet: There was an unexpected network error while writing to 192.168.204.52: rc = -22 [ 1161.338106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1161.392584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1161.443852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1161.485597] LNet: 59019:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1161.511420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1161.570883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1161.630500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1161.682835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1161.719259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1161.757748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1161.794793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1162.288376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -e local_dropped -r 1 [ 1162.330630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -e local_dropped -r 1 [ 1162.393882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.152@tcp [ 1162.421599] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 1162.666307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1162.695330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1162.760860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1162.811398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1162.851929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1162.891835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1162.928361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1162.996982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1163.034882] LNet: 59128:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1163.062608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1163.089877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1163.116904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1163.142656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1163.168335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1163.195991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1163.220648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1163.666873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -e local_aborted -r 1 [ 1163.700708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -e local_aborted -r 1 [ 1163.726816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.152@tcp [ 1163.981773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1164.032190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1164.062180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1164.107871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1164.155790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1164.212780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1164.264433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1164.309819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1164.326756] LNet: 59237:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1164.347143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1164.402913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1164.433625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1164.463971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1164.500025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1164.544112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1164.587596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1165.010921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -e local_no_route -r 1 [ 1165.046760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -e local_no_route -r 1 [ 1165.086554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.152@tcp [ 1165.113260] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 1165.122916] LNet: Skipped 1 previous similar message [ 1165.338215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1165.372782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1165.412657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1165.444937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1165.481909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1165.512532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1165.536893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1165.573471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1165.610196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1165.650546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1165.684572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1165.730359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1165.780852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1165.815274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1165.850532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1166.339132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -e local_timeout -r 1 [ 1166.380072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -e local_timeout -r 1 [ 1166.417366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.152@tcp [ 1166.656859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1166.688250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1166.714131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1166.744492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1166.772509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1166.809268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1166.846817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1166.876994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1166.898743] LNet: 59456:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1166.906867] LNet: 59456:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1166.929485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1166.976148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1167.013505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1167.039290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1167.075728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1167.127612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1167.169806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1167.698540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -e local_error -r 1 [ 1167.730240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -e local_error -r 1 [ 1167.764526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.152@tcp [ 1168.047656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1168.088258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1168.144916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1168.199457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1168.249531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1168.307893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1168.363893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1168.442976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1168.504547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1168.554327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1168.600417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1168.655843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1168.704935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1168.746467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1168.791601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1169.124651] LNet: 59610:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1169.135213] LNetError: 59610:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1169.160727] LNet: Removed LNI 192.168.204.52@tcp [ 1169.674481] Key type .llcrypt unregistered [ 1169.680451] Key type ._llcrypt unregistered [ 1191.456413] Key type ._llcrypt registered [ 1191.459148] Key type .llcrypt registered [ 1192.778677] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 07:12:57 (1787569977) [ 1193.649345] Key type .llcrypt unregistered [ 1193.652231] Key type ._llcrypt unregistered [ 1201.303733] Key type ._llcrypt registered [ 1201.305577] Key type .llcrypt registered [ 1201.559966] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1201.600868] alg: No test for adler32 (adler32-zlib) [ 1202.582637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1202.614991] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1211.096535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.152@tcp [ 1222.486903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1222.530867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1222.569118] LNet: Added LNI 192.168.204.52@tcp1 [8/256/0/180] [ 1223.851553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -e local_interrupt -r 1 [ 1223.881484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp1 -m GET -e local_interrupt -r 1 [ 1223.913898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -e local_interrupt -r 1 [ 1223.945534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp1 -m GET -e local_interrupt -r 1 [ 1223.979684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp -m GET -e local_interrupt -r 1 [ 1224.019426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp1 -m GET -e local_interrupt -r 1 [ 1224.073448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp -m GET -e local_interrupt -r 1 [ 1224.132720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp1 -m GET -e local_interrupt -r 1 [ 1224.193729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.152@tcp [ 1224.220377] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 1224.231137] LNet: 1 local NIs in recovery (showing 1): 192.168.204.52@tcp [ 1224.485406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1224.519501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1224.550996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1224.608449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1224.648650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1224.696199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1224.728336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1224.747915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1224.763335] LNet: 61082:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1224.777743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1224.804036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1224.830250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1224.853920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1224.891889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1224.917887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1224.956247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1225.449505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -e local_dropped -r 1 [ 1225.510616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp1 -m GET -e local_dropped -r 1 [ 1225.548947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -e local_dropped -r 1 [ 1225.594262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp1 -m GET -e local_dropped -r 1 [ 1225.629543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp -m GET -e local_dropped -r 1 [ 1225.662667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp1 -m GET -e local_dropped -r 1 [ 1225.698753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp -m GET -e local_dropped -r 1 [ 1225.749888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp1 -m GET -e local_dropped -r 1 [ 1225.804636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.152@tcp [ 1225.826330] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 1225.834696] LNet: Skipped 2 previous similar messages [ 1226.139407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1226.180758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1226.212225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1226.248399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1226.277833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1226.322830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1226.373127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1226.423819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1226.451618] LNet: 61204:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1226.481449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1226.519624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1226.565448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1226.625137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1226.688580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1226.751766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1226.790616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1227.440482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -e local_aborted -r 1 [ 1227.497247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp1 -m GET -e local_aborted -r 1 [ 1227.540674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -e local_aborted -r 1 [ 1227.581459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp1 -m GET -e local_aborted -r 1 [ 1227.613639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp -m GET -e local_aborted -r 1 [ 1227.644783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp1 -m GET -e local_aborted -r 1 [ 1227.678222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp -m GET -e local_aborted -r 1 [ 1227.721892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp1 -m GET -e local_aborted -r 1 [ 1227.780653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.152@tcp [ 1227.805216] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 1227.813687] LNet: Skipped 2 previous similar messages [ 1228.067021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1228.101265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1228.126557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1228.148844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1228.176223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1228.203889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1228.246943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1228.275173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1228.299504] LNet: 61325:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1228.319566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1228.361279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1228.405564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1228.446319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1228.485235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1228.510292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1228.537238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1229.117567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -e local_no_route -r 1 [ 1229.153804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp1 -m GET -e local_no_route -r 1 [ 1229.199873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -e local_no_route -r 1 [ 1229.234872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp1 -m GET -e local_no_route -r 1 [ 1229.266292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp -m GET -e local_no_route -r 1 [ 1229.299968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp1 -m GET -e local_no_route -r 1 [ 1229.336898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp -m GET -e local_no_route -r 1 [ 1229.365383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp1 -m GET -e local_no_route -r 1 [ 1229.397947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.152@tcp [ 1229.746722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1229.771632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1229.795665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1229.858369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1229.925868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1229.986394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1230.045684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1230.098157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1230.138625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1230.171484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1230.197807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1230.224983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1230.254889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1230.285535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1230.313968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1230.824144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -e local_timeout -r 1 [ 1230.857312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp1 -m GET -e local_timeout -r 1 [ 1230.891743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -e local_timeout -r 1 [ 1230.924771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp1 -m GET -e local_timeout -r 1 [ 1230.955619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp -m GET -e local_timeout -r 1 [ 1231.000974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp1 -m GET -e local_timeout -r 1 [ 1231.058884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp -m GET -e local_timeout -r 1 [ 1231.157156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp1 -m GET -e local_timeout -r 1 [ 1231.203425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.152@tcp [ 1231.240669] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 1231.248235] LNet: Skipped 5 previous similar messages [ 1231.543338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1231.585546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1231.636253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1231.668589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1231.714867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1231.773887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1231.817773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1231.856550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1231.888305] LNet: 61567:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1231.894174] LNet: 61567:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1231.918739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1231.953825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1231.996222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1232.059211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1232.107281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1232.168566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1232.249569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1232.758136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -e local_error -r 1 [ 1232.800796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp1 -m GET -e local_error -r 1 [ 1232.854503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -e local_error -r 1 [ 1232.917244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp1 -m GET -e local_error -r 1 [ 1232.970831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp -m GET -e local_error -r 1 [ 1233.016107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp1 -m GET -e local_error -r 1 [ 1233.052875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp -m GET -e local_error -r 1 [ 1233.080738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp1 -m GET -e local_error -r 1 [ 1233.109712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.152@tcp [ 1233.299654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1233.329727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1233.356503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1233.378303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1233.402963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1233.431256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1233.461339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1233.489088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1233.516729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1233.543477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1233.568955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1233.598834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1233.631909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1233.658700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1233.690706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1239.916628] LNet: 61768:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1239.927945] LNetError: 61768:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1239.939281] LNet: Removed LNI 192.168.204.52@tcp [ 1239.986169] LNet: Removed LNI 192.168.204.52@tcp1 [ 1240.523926] Key type .llcrypt unregistered [ 1240.528559] Key type ._llcrypt unregistered [ 1256.774437] Key type ._llcrypt registered [ 1256.776538] Key type .llcrypt registered [ 1258.255619] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 07:14:03 (1787570043) [ 1259.361246] Key type .llcrypt unregistered [ 1259.367144] Key type ._llcrypt unregistered [ 1266.730858] Key type ._llcrypt registered [ 1266.733393] Key type .llcrypt registered [ 1266.910925] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1266.936445] alg: No test for adler32 (adler32-zlib) [ 1267.937637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1268.014155] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1276.354800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.152@tcp [ 1277.467070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -e remote_dropped -r 1 [ 1277.519915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -e remote_dropped -r 1 [ 1277.559048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.152@tcp [ 1277.592110] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 1277.964492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1277.999479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1278.027970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1278.066603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1278.103227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1278.183896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1278.233254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1278.274100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1278.302608] LNet: 63035:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1278.325548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1278.364106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1278.415132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1278.477867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1278.545627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1278.589152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1278.620727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1279.052208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -e remote_error -r 1 [ 1279.087244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -e remote_error -r 1 [ 1279.122753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.152@tcp [ 1279.143711] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 1279.349463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1279.377674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1279.404957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1279.435442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1279.470981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1279.514231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1279.549063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1279.587448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1279.613151] LNet: 63144:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1279.635528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1279.689729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1279.719877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1279.744790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1279.770119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1279.792936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1279.828328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1280.309891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -e remote_timeout -r 1 [ 1280.340808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -e remote_timeout -r 1 [ 1280.368543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.152@tcp [ 1280.384950] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 1280.601131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1280.623336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1280.648906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1280.681330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1280.717297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1280.751605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1280.781231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1280.819210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1280.839443] LNet: 63253:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1280.867925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1280.920864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1280.958899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1280.992398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1281.024180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1281.073921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1281.132643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1281.464038] LNet: 63299:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1281.471804] LNetError: 63299:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1281.493862] LNet: Removed LNI 192.168.204.52@tcp [ 1281.960417] Key type .llcrypt unregistered [ 1281.965511] Key type ._llcrypt unregistered [ 1298.504642] Key type ._llcrypt registered [ 1298.506964] Key type .llcrypt registered [ 1299.819157] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 07:14:44 (1787570084) [ 1300.912259] Key type .llcrypt unregistered [ 1300.914926] Key type ._llcrypt unregistered [ 1307.841505] Key type ._llcrypt registered [ 1307.847458] Key type .llcrypt registered [ 1308.143682] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1308.172259] alg: No test for adler32 (adler32-zlib) [ 1309.127992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1309.171868] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1318.348692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.152@tcp [ 1328.071671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1328.131615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1328.172107] LNet: Added LNI 192.168.204.52@tcp1 [8/256/0/180] [ 1329.549343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -e remote_dropped -r 1 [ 1329.595987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp1 -m GET -e remote_dropped -r 1 [ 1329.629511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -e remote_dropped -r 1 [ 1329.662813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp1 -m GET -e remote_dropped -r 1 [ 1329.700351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp -m GET -e remote_dropped -r 1 [ 1329.738773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp1 -m GET -e remote_dropped -r 1 [ 1329.775591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp -m GET -e remote_dropped -r 1 [ 1329.821225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp1 -m GET -e remote_dropped -r 1 [ 1329.864747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.152@tcp [ 1329.890791] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 1329.896800] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.152@tcp [ 1330.075488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1330.106337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1330.141172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1330.173244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1330.204616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1330.233915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1330.268365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1330.317572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1330.338637] LNet: 64779:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1330.364802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1330.418981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1330.456098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1330.493312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1330.536120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1330.580306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1330.628121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1330.656818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1331.255898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -e remote_error -r 1 [ 1331.319796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp1 -m GET -e remote_error -r 1 [ 1331.396347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -e remote_error -r 1 [ 1331.449444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp1 -m GET -e remote_error -r 1 [ 1331.496784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp -m GET -e remote_error -r 1 [ 1331.547356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp1 -m GET -e remote_error -r 1 [ 1331.594257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp -m GET -e remote_error -r 1 [ 1331.654472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp1 -m GET -e remote_error -r 1 [ 1331.699559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.152@tcp [ 1331.719877] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 1331.729760] LNet: Skipped 2 previous similar messages [ 1331.992865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1332.024754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1332.065680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1332.111340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1332.165925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1332.213573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1332.255821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1332.304190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1332.347728] LNet: 64902:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1332.382424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1332.438938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1332.473890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1332.512604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1332.557818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1332.599920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1332.642093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1332.677949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1333.217901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -e remote_timeout -r 1 [ 1333.268836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp1 -m GET -e remote_timeout -r 1 [ 1333.303558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -e remote_timeout -r 1 [ 1333.339049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp1 -m GET -e remote_timeout -r 1 [ 1333.381176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp -m GET -e remote_timeout -r 1 [ 1333.432169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp1 -m GET -e remote_timeout -r 1 [ 1333.479722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp -m GET -e remote_timeout -r 1 [ 1333.532682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp1 -m GET -e remote_timeout -r 1 [ 1333.576703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.152@tcp [ 1333.602491] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 1333.892944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1333.919926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1333.950337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1333.974979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1334.010387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1334.035759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1334.059603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1334.085146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1334.099939] LNet: 65025:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1334.115991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1334.149762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1334.193160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1334.225970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1334.263885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1334.298565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1334.353846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1334.418845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1339.835365] LNet: 65107:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1339.844425] LNetError: 65107:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1339.850598] LNet: Removed LNI 192.168.204.52@tcp [ 1339.869259] LNet: Removed LNI 192.168.204.52@tcp1 [ 1340.452811] Key type .llcrypt unregistered [ 1340.455978] Key type ._llcrypt unregistered [ 1359.699651] Key type ._llcrypt registered [ 1359.702221] Key type .llcrypt registered [ 1361.216961] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 07:15:45 (1787570145) [ 1362.888677] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1365.315063] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 07:15:49 (1787570149) [ 1366.366121] Key type .llcrypt unregistered [ 1366.369643] Key type ._llcrypt unregistered [ 1373.760893] Key type ._llcrypt registered [ 1373.768342] Key type .llcrypt registered [ 1373.995988] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1374.016441] alg: No test for adler32 (adler32-zlib) [ 1375.008719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1375.044390] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1382.937878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.152@tcp [ 1383.659218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -e network_timeout -r 1 [ 1383.695764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -e network_timeout -r 1 [ 1383.724195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.152@tcp [ 1383.744284] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 1383.950368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1383.983455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1384.024446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1384.057212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1384.116804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1384.155681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1384.195801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1384.257831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1384.287301] LNet: 1 local NIs in recovery (showing 1): 192.168.204.52@tcp [ 1384.291625] LNet: 66655:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1384.293411] LNet: There was an unexpected network error while writing to 192.168.204.52: rc = -22 [ 1384.330691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1384.369689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1384.397975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1384.429153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1384.464681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1384.515857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1384.795839] LNet: 66698:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1384.801161] LNetError: 66698:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1385.830434] LNet: Removed LNI 192.168.204.52@tcp [ 1386.296601] Key type .llcrypt unregistered [ 1386.302953] Key type ._llcrypt unregistered [ 1406.690917] Key type ._llcrypt registered [ 1406.694761] Key type .llcrypt registered [ 1406.929172] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1406.962901] alg: No test for adler32 (adler32-zlib) [ 1407.848372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1407.899373] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1416.483911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.152@tcp [ 1426.414479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1426.450252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1426.469079] LNet: Added LNI 192.168.204.52@tcp1 [8/256/0/180] [ 1427.999962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -e network_timeout -r 1 [ 1428.054820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp1 -m GET -e network_timeout -r 1 [ 1428.129439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -e network_timeout -r 1 [ 1428.209949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp1 -m GET -e network_timeout -r 1 [ 1428.301767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp -m GET -e network_timeout -r 1 [ 1428.346926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp1 -m GET -e network_timeout -r 1 [ 1428.398450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp -m GET -e network_timeout -r 1 [ 1428.442927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp1 -m GET -e network_timeout -r 1 [ 1428.498405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.152@tcp [ 1428.538371] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 1428.898200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1428.930768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1428.963198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1428.995602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1429.033330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1429.063856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1429.120416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1429.187736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1429.212411] LNet: 67582:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1429.257218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1429.292388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1429.344593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1429.409171] LNet: 1 local NIs in recovery (showing 1): 192.168.204.52@tcp [ 1429.410134] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 1429.422091] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.152@tcp [ 1429.460269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1429.467679] LNet: Skipped 1 previous similar message [ 1429.553772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1429.600996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1435.347286] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 1435.772652] LNet: 67660:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1435.788829] LNetError: 67660:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1435.809465] LNet: Removed LNI 192.168.204.52@tcp [ 1435.840949] LNet: Removed LNI 192.168.204.52@tcp1 [ 1436.314578] Key type .llcrypt unregistered [ 1436.317504] Key type ._llcrypt unregistered [ 1453.953840] Key type ._llcrypt registered [ 1453.959732] Key type .llcrypt registered [ 1455.567235] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 07:17:20 (1787570240) [ 1456.224331] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1456.254585] alg: No test for adler32 (adler32-zlib) [ 1457.218146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1457.289628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1457.330824] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1457.962770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1457.993578] LNet: Added LNI 192.168.204.52@tcp1 [8/256/0/180] [ 1458.735786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.52@tcp [ 1458.816495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1458.862831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1458.907891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1458.971479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1459.024307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1459.079674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.52@tcp [ 1459.107193] LNetError: 68760:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.204.52@tcp to recovery [ 1459.295376] LNet: 1 local NIs in recovery (showing 1): 192.168.204.52@tcp [ 1459.296610] LNet: There was an unexpected network error while writing to 192.168.204.52: rc = -22 [ 1461.368884] LNet: There was an unexpected network error while writing to 192.168.204.52: rc = -22 [ 1461.638575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1465.503327] LNet: There was an unexpected network error while writing to 192.168.204.52: rc = -22 [ 1466.425897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1466.498769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1466.595324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1466.615328] LNet: 68867:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1466.620269] LNetError: 68867:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1466.625249] LNet: Removed LNI 192.168.204.52@tcp [ 1466.631752] LNet: Removed LNI 192.168.204.52@tcp1 [ 1466.649542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1466.738689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1466.770601] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1467.491235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1467.513538] LNet: Added LNI 192.168.204.52@tcp1 [8/256/0/180] [ 1468.443493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.52@tcp [ 1468.489576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1468.554834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1468.622062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1468.713822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1468.766768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1468.833750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1468.910260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.52@tcp [ 1468.955416] LNetError: 69140:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.204.52@tcp to recovery [ 1469.791509] LNet: There was an unexpected network error while writing to 192.168.204.52: rc = -22 [ 1472.593219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1475.936082] LNet: There was an unexpected network error while writing to 192.168.204.52: rc = -22 [ 1475.956762] LNet: Skipped 1 previous similar message [ 1479.620612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1479.667837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1479.737746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1486.532766] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 07:17:51 (1787570271) [ 1486.701876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1486.733675] LNet: 69846:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1486.749427] LNetError: 69846:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1486.756259] LNetError: 69846:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1486.764901] LNet: Removed LNI 192.168.204.52@tcp [ 1486.788386] LNet: Removed LNI 192.168.204.52@tcp1 [ 1486.820908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1486.885607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1486.917357] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1487.703493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1487.750326] LNet: Added LNI 192.168.204.52@tcp1 [8/256/0/180] [ 1488.819679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.52@tcp [ 1488.898795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1488.931914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1488.976122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1489.043875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1489.085392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1489.139881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.52@tcp --health 0 [ 1489.269434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1489.953058] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.52@tcp [ 1490.975670] LNet: There was an unexpected network error while writing to 192.168.204.52: rc = -22 [ 1490.985567] LNet: Skipped 1 previous similar message [ 1505.735125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1505.910707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1505.954448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1506.014673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.52@tcp --health 500 [ 1506.202814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1512.912985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1512.972990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1512.992863] LNet: 70346:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1513.002467] LNetError: 70346:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1513.009404] LNetError: 70346:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1513.018626] LNet: Removed LNI 192.168.204.52@tcp [ 1513.031263] LNet: Removed LNI 192.168.204.52@tcp1 [ 1513.068398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1513.133435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1513.170364] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1513.727407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1513.741659] LNet: Added LNI 192.168.204.52@tcp1 [8/256/0/180] [ 1514.508561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.52@tcp [ 1514.554316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1514.606685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1514.631556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1514.677144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1514.709592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1514.739738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1514.765592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.52@tcp --health 0 [ 1515.168449] LNet: There was an unexpected network error while writing to 192.168.204.52: rc = -22 [ 1515.178280] LNet: Skipped 3 previous similar messages [ 1516.162361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1526.602290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1526.628697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1526.654160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1534.681802] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 07:18:39 (1787570319) [ 1535.410456] LNet: 71393:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1535.418326] LNetError: 71393:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1535.430252] LNetError: 71393:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1535.442283] LNet: Removed LNI 192.168.204.52@tcp [ 1535.490247] LNet: Removed LNI 192.168.204.52@tcp1 [ 1536.360804] Key type .llcrypt unregistered [ 1536.363090] Key type ._llcrypt unregistered [ 1544.292571] Key type ._llcrypt registered [ 1544.294920] Key type .llcrypt registered [ 1544.474884] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1544.484884] alg: No test for adler32 (adler32-zlib) [ 1545.434142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1545.482086] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1555.044529] Lustre: DEBUG MARKER: Initial discovery [ 1555.099521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.152@tcp [ 1558.026952] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1558.282554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1558.319291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.152@tcp [ 1558.359536] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 1558.380315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1558.419566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.204.152@tcp -r 1 -m GET -l 3 [ 1558.462345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.204.152@tcp -r 1 -m GET -e local_error [ 1558.490766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.152@tcp -r 1 -m PUT -l 6 [ 1558.815759] LNet: 1 local NIs in recovery (showing 1): 192.168.204.52@tcp [ 1559.845910] Lustre: DEBUG MARKER: Force oleg452-client.virtnet to discover 192.168.204.152@tcp (in background) [ 1560.152631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.152@tcp [ 1561.378394] Lustre: DEBUG MARKER: Force oleg452-server to discover 192.168.204.52@tcp [ 1563.233378] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 1563.843872] Lustre: DEBUG MARKER: Wait for 72161 [ 1565.440097] Lustre: DEBUG MARKER: Finished wait on 72161 [ 1565.624681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1565.665853] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 1565.682389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1566.008540] LNet: 72614:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1566.015696] LNetError: 72614:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1566.041230] LNet: Removed LNI 192.168.204.52@tcp [ 1566.403507] Key type .llcrypt unregistered [ 1566.405451] Key type ._llcrypt unregistered [ 1585.583622] Key type ._llcrypt registered [ 1585.585930] Key type .llcrypt registered [ 1587.448553] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 07:19:31 (1787570371) [ 1588.081708] Key type .llcrypt unregistered [ 1588.084262] Key type ._llcrypt unregistered [ 1597.100983] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1597.262316] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1597.264981] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1597.272422] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1600.094847] Key type ._llcrypt registered [ 1600.096546] Key type .llcrypt registered [ 1600.377172] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1600.397539] alg: No test for adler32 (adler32-zlib) [ 1601.285347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1601.330305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1601.361937] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1601.919712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1601.937332] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1612.108737] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 07:19:56 (1787570396) [ 1612.425864] LNet: 74697:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1612.434968] LNetError: 74697:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1612.449333] LNet: Removed LNI 10.1.2.3@tcp [ 1612.469248] LNet: Removed LNI 192.168.204.52@tcp [ 1613.244525] Key type .llcrypt unregistered [ 1613.248803] Key type ._llcrypt unregistered [ 1619.398507] Key type ._llcrypt registered [ 1619.403336] Key type .llcrypt registered [ 1619.678900] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1619.745401] alg: No test for adler32 (adler32-zlib) [ 1620.656303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1620.784439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1620.816525] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1621.750377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1621.772233] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1622.874433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.204.52@tcp [ 1623.968981] LNetError: Unexpected error -22 connecting to 192.168.204.52@tcp at host 192.168.204.52:988 [ 1630.385657] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 07:20:15 (1787570415) [ 1631.066145] LNet: 75832:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1631.075289] LNetError: 75832:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1631.082877] LNet: Removed LNI 10.1.2.3@tcp [ 1631.094117] LNet: Removed LNI 192.168.204.52@tcp [ 1631.958361] Key type .llcrypt unregistered [ 1631.960229] Key type ._llcrypt unregistered [ 1638.724440] Key type ._llcrypt registered [ 1638.727155] Key type .llcrypt registered [ 1639.040995] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1639.054277] alg: No test for adler32 (adler32-zlib) [ 1640.074035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1640.156917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1640.176562] LNet: Added LNI 192.168.204.52@tcp1 [8/256/0/180] [ 1640.813060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1640.859820] LNet: Added LNI 192.168.204.52@tcp2 [8/256/0/180] [ 1642.003638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.204.52@tcp1 --nid 192.168.204.52@tcp2 [ 1646.971572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.52@tcp1 192.168.204.52@tcp2 [ 1646.987162] LNetError: 76672:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.204.52@tcp1 to 192.168.204.52@tcp2 - Route table may be misconfigured [ 1647.011994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.52@tcp2 192.168.204.52@tcp1 [ 1654.260631] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 07:20:38 (1787570438) [ 1654.437762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1654.478519] LNet: 77242:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1654.500086] LNetError: 77242:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1654.513681] LNet: Removed LNI 192.168.204.52@tcp1 [ 1655.527248] LNet: Removed LNI 192.168.204.52@tcp2 [ 1655.588123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1655.678073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1655.729141] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1656.833823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1656.863252] LNet: Added LNI 192.168.204.52@tcp1 [8/256/0/180] [ 1657.756815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.52@tcp [ 1657.781837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.52@tcp 192.168.204.52@tcp [ 1657.808266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.52@tcp1 192.168.204.52@tcp1 [ 1657.842959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.52@tcp -d 192.168.204.52@tcp -e network_timeout [ 1657.878967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.52@tcp -d 192.168.204.52@tcp1 -e network_timeout [ 1657.914922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp -e network_timeout [ 1657.951095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp1 -e network_timeout [ 1657.989297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 1658.005961] LNet: There was an unexpected network error while writing to 192.168.204.52: rc = -22 [ 1658.037136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1658.127453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1658.223849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1658.727880] LNet: 1 local NIs in recovery (showing 1): 192.168.204.52@tcp [ 1665.338184] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 07:20:49 (1787570449) [ 1665.523855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1665.570712] LNet: 78108:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1665.578966] LNetError: 78108:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1665.587291] LNetError: 78108:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1665.594654] LNet: Removed LNI 192.168.204.52@tcp [ 1666.668490] LNet: Removed LNI 192.168.204.52@tcp1 [ 1666.723261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1666.853070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1667.222072] LNet: 78149:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1667.630534] Key type .llcrypt unregistered [ 1667.632612] Key type ._llcrypt unregistered [ 1679.788851] Key type ._llcrypt registered [ 1679.792233] Key type .llcrypt registered [ 1681.238283] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 07:21:06 (1787570466) [ 1682.223368] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1682.249543] alg: No test for adler32 (adler32-zlib) [ 1683.248579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1685.010311] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1687.079101] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 07:21:11 (1787570471) [ 1687.250693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1687.292304] LNet: 79244:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1687.346346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1687.419662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1687.441629] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1688.181939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1688.208951] LNet: Added LNI 192.168.204.52@tcp1 [8/256/0/180] [ 1689.264772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 1689.289777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp1 [ 1689.314175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.52@tcp1 [ 1697.260888] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 07:21:21 (1787570481) [ 1699.146284] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg452-server [ 1701.119084] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 07:21:25 (1787570485) [ 1703.154939] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg452-server [ 1705.417699] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 07:21:30 (1787570490) [ 1707.238877] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg452-server [ 1709.233686] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 07:21:34 (1787570494) [ 1711.430836] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg452-server [ 1713.807921] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 07:21:38 (1787570498) [ 1716.096267] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg452-server [ 1718.506008] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 07:21:42 (1787570502) [ 1720.772345] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg452-server [ 1722.838448] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 07:21:47 (1787570507) [ 1724.713852] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg452-server [ 1726.899358] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 07:21:51 (1787570511) [ 1729.088250] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg452-server [ 1731.337478] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 07:21:55 (1787570515) [ 1733.458865] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg452-server [ 1735.542962] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 07:22:00 (1787570520) [ 1736.112962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1736.154902] LNet: 83125:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1736.169931] LNetError: 83125:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1736.186964] LNet: Removed LNI 192.168.204.52@tcp [ 1737.256853] LNet: Removed LNI 192.168.204.52@tcp1 [ 1737.290231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1737.355381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1737.407059] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1738.330432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1738.406914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 1738.605465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1738.626798] LNet: 83272:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1738.637853] LNetError: 83272:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1738.649018] LNetError: 83272:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1738.661388] LNet: Removed LNI 192.168.204.52@tcp [ 1738.689309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1738.785047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1738.808217] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1739.831170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 1740.034170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1740.079785] LNetError: 83417:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1741.095831] LNet: Removed LNI 192.168.204.52@tcp [ 1741.140292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1741.236309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1741.277586] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1742.172221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1742.312261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 1742.520162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1742.540469] LNet: 83565:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1742.559903] LNet: 83565:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1742.574993] LNetError: 83565:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1742.606886] LNet: Removed LNI 192.168.204.52@tcp [ 1742.637781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1742.771941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1742.820736] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1744.316601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 1744.612392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1745.703911] LNet: Removed LNI 192.168.204.52@tcp [ 1745.752284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1745.849184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1745.877093] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1746.832145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1746.919930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 1747.116443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1747.169663] LNet: 83858:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1747.184820] LNet: 83858:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1747.199048] LNetError: 83858:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1747.214792] LNetError: 83858:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1747.247187] LNet: Removed LNI 192.168.204.52@tcp [ 1747.287884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1747.360887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1747.404831] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1748.303794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 1748.584264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1749.672347] LNet: Removed LNI 192.168.204.52@tcp [ 1749.767767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1749.868994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1749.896182] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1750.894886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1750.979811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 1752.479379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1753.574770] LNet: Removed LNI 192.168.204.52@tcp [ 1753.631364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1753.715754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1753.753384] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1754.610655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 1754.930029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1756.016463] LNet: Removed LNI 192.168.204.52@tcp [ 1756.099813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1756.177586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1756.226288] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1757.199762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1757.288414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 1757.638562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1757.658812] LNet: 84453:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1757.674754] LNet: 84453:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1757.692959] LNetError: 84453:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1757.700638] LNetError: 84453:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1758.761450] LNet: Removed LNI 192.168.204.52@tcp [ 1758.817900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1758.891136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1758.932126] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1759.848447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 1760.189867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1761.259270] LNet: Removed LNI 192.168.204.52@tcp [ 1761.341930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1761.473690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1761.502765] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1762.679460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1762.793540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 1763.097912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1764.199328] LNet: Removed LNI 192.168.204.52@tcp [ 1764.216362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1764.271328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1764.314808] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1765.220252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 1765.484764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1766.569048] LNet: Removed LNI 192.168.204.52@tcp [ 1766.592961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1766.657136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1766.679889] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1768.067395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1768.153664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1768.194271] LNet: Removed LNI 192.168.204.52@tcp [ 1768.255958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1768.326405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1768.392364] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1769.142354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1769.213269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1769.251735] LNet: Removed LNI 192.168.204.52@tcp [ 1769.281653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1769.411319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1769.584057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1769.630836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1769.682855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1769.779705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1769.842890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1769.947572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1770.108602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1770.174909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1770.274803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1770.325986] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1771.387976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1771.435784] LNet: Removed LNI 192.168.204.52@tcp [ 1771.463205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1771.575291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1780.584205] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 07:22:45 (1787570565) [ 1780.896948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1780.942549] LNet: 85913:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1780.964869] LNet: 85913:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 1781.019552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1781.127984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1781.180292] LNet: Added LNI 192.168.204.52@tcp231 [8/256/0/180] [ 1783.121590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1783.149334] LNetError: 86053:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1783.155966] LNetError: 86053:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1783.171034] LNet: Removed LNI 192.168.204.52@tcp231 [ 1783.231536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1783.368446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1783.428598] LNet: Added LNI 192.168.204.52@tcp231 [8/256/0/180] [ 1783.434500] LNet: 86061:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1784.215854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1784.258770] LNet: Removed LNI 192.168.204.52@tcp231 [ 1784.304676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1784.358347] LNet: Added LNI 192.168.204.52@tcp231 [8/256/0/180] [ 1786.227599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1786.295032] LNet: Removed LNI 192.168.204.52@tcp231 [ 1786.350295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1786.480342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1786.517410] LNet: Added LNI 192.168.204.52@tcp231 [8/256/0/0] [ 1786.539446] LNet: 86213:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1787.428552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1787.498739] LNet: Removed LNI 192.168.204.52@tcp231 [ 1787.560056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1787.603388] LNet: Added LNI 192.168.204.52@tcp231 [8/256/0/0] [ 1789.682757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1789.734909] LNet: Removed LNI 192.168.204.52@tcp231 [ 1789.783438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1789.958495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1790.048343] LNet: Added LNI 192.168.204.52@tcp231 [8/256/0/60] [ 1790.065499] LNet: 86365:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1790.738797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1790.781533] LNet: Removed LNI 192.168.204.52@tcp231 [ 1790.797862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1790.828983] LNet: Added LNI 192.168.204.52@tcp231 [8/256/0/60] [ 1792.399053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1792.427354] LNet: Removed LNI 192.168.204.52@tcp231 [ 1792.467894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1792.609154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1792.664863] LNet: Added LNI 192.168.204.52@tcp231 [8/256/0/180] [ 1792.681295] LNet: 86517:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1793.396766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1793.446732] LNet: Removed LNI 192.168.204.52@tcp231 [ 1793.470957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1793.519593] LNet: Added LNI 192.168.204.52@tcp231 [8/256/0/180] [ 1804.024827] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 07:23:08 (1787570588) [ 1804.165597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1804.195245] LNet: Removed LNI 192.168.204.52@tcp231 [ 1804.214589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1804.291885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1804.325100] LNet: Added LNI 192.168.204.52@tcp232 [8/256/0/180] [ 1804.966201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1804.999343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1805.036217] LNet: Removed LNI 192.168.204.52@tcp232 [ 1805.053579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1805.095923] LNet: Added LNI 192.168.204.52@tcp232 [8/256/0/180] [ 1805.719931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1805.749566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1805.778555] LNet: Removed LNI 192.168.204.52@tcp232 [ 1805.791944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1805.822896] LNet: Added LNI 192.168.204.52@tcp232 [8/256/0/180] [ 1806.439680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1806.472655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1806.502937] LNet: Removed LNI 192.168.204.52@tcp232 [ 1806.519907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1806.539540] LNet: Added LNI 192.168.204.52@tcp232 [8/256/0/180] [ 1807.398068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1807.438173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1807.492780] LNet: Removed LNI 192.168.204.52@tcp232 [ 1807.522695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1814.420584] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 07:23:19 (1787570599) [ 1814.905564] LNet: 88355:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1814.909444] LNet: 88355:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 1815.628760] Key type .llcrypt unregistered [ 1815.637730] Key type ._llcrypt unregistered [ 1822.999548] Key type ._llcrypt registered [ 1823.007349] Key type .llcrypt registered [ 1823.351509] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1823.401121] alg: No test for adler32 (adler32-zlib) [ 1824.422466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1824.447725] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1833.025798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.152@tcp [ 1843.872699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1843.934930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1843.976735] LNet: Added LNI 192.168.204.52@tcp1 [8/256/0/180] [ 1845.916881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -e local_interrupt -r 2 [ 1845.956165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp1 -m GET -e local_interrupt -r 2 [ 1845.996358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -e local_interrupt -r 2 [ 1846.043769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp1 -m GET -e local_interrupt -r 2 [ 1846.093247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp -m GET -e local_interrupt -r 2 [ 1846.145475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp1 -m GET -e local_interrupt -r 2 [ 1846.216756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp -m GET -e local_interrupt -r 2 [ 1846.273762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp1 -m GET -e local_interrupt -r 2 [ 1846.315447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.52@tcp 192.168.204.152@tcp [ 1846.349404] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 1846.354867] LNet: 1 local NIs in recovery (showing 1): 192.168.204.52@tcp [ 1846.754235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1846.792331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1846.838469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1846.878508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1846.948320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1846.991748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1847.057441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1847.126466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1847.155717] LNet: 89023:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1847.173333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1847.201555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1847.234566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1847.270980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1847.301771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1847.331432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1848.126646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -e local_dropped -r 2 [ 1848.168761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp1 -m GET -e local_dropped -r 2 [ 1848.236925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -e local_dropped -r 2 [ 1848.287225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp1 -m GET -e local_dropped -r 2 [ 1848.346725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp -m GET -e local_dropped -r 2 [ 1848.406743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp1 -m GET -e local_dropped -r 2 [ 1848.480950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp -m GET -e local_dropped -r 2 [ 1848.560947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp1 -m GET -e local_dropped -r 2 [ 1848.625077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.52@tcp 192.168.204.152@tcp [ 1848.658874] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 1848.954549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1849.007263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1849.045077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1849.100730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1849.144507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1849.171980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1849.198835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1849.223826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1849.244414] LNet: 89144:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1849.263528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1849.294351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1849.321132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1849.354143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1849.386343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1849.424889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1850.086288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -e local_aborted -r 2 [ 1850.128654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp1 -m GET -e local_aborted -r 2 [ 1850.166168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -e local_aborted -r 2 [ 1850.218202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp1 -m GET -e local_aborted -r 2 [ 1850.261831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp -m GET -e local_aborted -r 2 [ 1850.297401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp1 -m GET -e local_aborted -r 2 [ 1850.351773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp -m GET -e local_aborted -r 2 [ 1850.399145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp1 -m GET -e local_aborted -r 2 [ 1850.448993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.52@tcp 192.168.204.152@tcp [ 1850.471913] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 1850.845564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1850.888130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1850.934905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1850.971401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1851.019261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1851.083446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1851.129698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1851.170113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1851.188836] LNet: 89264:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1851.203622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1851.244437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1851.294252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1851.329483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1851.392421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1851.427473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1852.119979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -e local_no_route -r 2 [ 1852.153263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp1 -m GET -e local_no_route -r 2 [ 1852.210991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -e local_no_route -r 2 [ 1852.271235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp1 -m GET -e local_no_route -r 2 [ 1852.327963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp -m GET -e local_no_route -r 2 [ 1852.389233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp1 -m GET -e local_no_route -r 2 [ 1852.445906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp -m GET -e local_no_route -r 2 [ 1852.485521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp1 -m GET -e local_no_route -r 2 [ 1852.526093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.52@tcp 192.168.204.152@tcp [ 1852.561410] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 1852.867700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1852.902744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1852.959889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1853.011148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1853.056251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1853.087846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1853.114870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1853.145164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1853.187583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1853.216672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1853.247711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1853.290067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1853.327185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1853.368914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1853.922285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -e local_timeout -r 2 [ 1854.026613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp1 -m GET -e local_timeout -r 2 [ 1854.088275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -e local_timeout -r 2 [ 1854.131510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp1 -m GET -e local_timeout -r 2 [ 1854.179139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp -m GET -e local_timeout -r 2 [ 1854.248279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp1 -m GET -e local_timeout -r 2 [ 1854.316576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp -m GET -e local_timeout -r 2 [ 1854.361293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp1 -m GET -e local_timeout -r 2 [ 1854.394916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.52@tcp 192.168.204.152@tcp [ 1854.787667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1854.827301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1854.890715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1854.938267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1854.991650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1855.031751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1855.074030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1855.120644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1855.143933] LNet: 89505:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1855.149708] LNet: 89505:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1855.166357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1855.203966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1855.253691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1855.321904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1855.410209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1855.469340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1856.207393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -e remote_dropped -r 2 [ 1856.261471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp1 -m GET -e remote_dropped -r 2 [ 1856.324574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -e remote_dropped -r 2 [ 1856.380176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp1 -m GET -e remote_dropped -r 2 [ 1856.410867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp -m GET -e remote_dropped -r 2 [ 1856.441964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp1 -m GET -e remote_dropped -r 2 [ 1856.489736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp -m GET -e remote_dropped -r 2 [ 1856.569042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp1 -m GET -e remote_dropped -r 2 [ 1856.660164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.52@tcp 192.168.204.152@tcp [ 1856.689285] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 1856.697170] LNet: Skipped 1 previous similar message [ 1856.701493] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.152@tcp [ 1856.947583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1856.973901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1857.001907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1857.044308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1857.095229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1857.133634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1857.182037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1857.213270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1857.272675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1857.323339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1857.374713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1857.423601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1857.465920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1857.506685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1863.922939] LNet: 89704:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1863.934272] LNetError: 89704:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1863.941243] LNet: Removed LNI 192.168.204.52@tcp [ 1863.972678] LNet: Removed LNI 192.168.204.52@tcp1 [ 1864.359777] Key type .llcrypt unregistered [ 1864.362788] Key type ._llcrypt unregistered [ 1881.661652] Key type ._llcrypt registered [ 1881.670546] Key type .llcrypt registered [ 1882.934034] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 07:24:27 (1787570667) [ 1883.577929] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1883.594145] alg: No test for adler32 (adler32-zlib) [ 1884.537204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1884.582357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1884.607917] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1885.727352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1885.753648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.52@tcp [ 1885.771632] LNetError: 90685:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.204.52@tcp to recovery [ 1885.869543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.204.52@tcp [ 1885.972561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1892.340161] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 07:24:37 (1787570677) [ 1892.442801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1892.480993] LNet: 91269:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1892.493112] LNetError: 91269:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1892.515951] LNet: Removed LNI 192.168.204.52@tcp [ 1892.534189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1892.602934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1892.639030] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1893.376848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1893.409740] LNet: Added LNI 192.168.204.52@tcp1 [8/256/0/180] [ 1894.396109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.204.52@tcp [ 1894.439397] LNetError: 91542:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.204.52@tcp to recovery [ 1894.623326] LNet: 1 local NIs in recovery (showing 1): 192.168.204.52@tcp [ 1894.627504] LNet: There was an unexpected network error while writing to 192.168.204.52: rc = -22 [ 1895.736505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.204.52@tcp [ 1903.532269] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 07:24:48 (1787570688) [ 1905.738516] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg452-server [ 1907.959576] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 07:24:52 (1787570692) [ 1910.436273] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg452-server [ 1912.198828] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 07:24:57 (1787570697) [ 1914.364391] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg452-server [ 1916.928871] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 07:25:01 (1787570701) [ 1917.118391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1917.171630] LNet: 93160:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1917.184314] LNetError: 93160:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1918.240661] LNet: Removed LNI 192.168.204.52@tcp [ 1918.247343] LNetError: 93160:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1918.268681] LNet: Removed LNI 192.168.204.52@tcp1 [ 1918.335572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1918.433795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1918.458233] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1919.817427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1920.095502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1920.199681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1920.396184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1920.493601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1920.681887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1927.887625] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 07:25:12 (1787570712) [ 1928.237168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1928.267647] LNet: 93897:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1928.274451] LNetError: 93897:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1928.301240] LNet: Removed LNI 192.168.204.52@tcp [ 1928.349547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1928.436436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1928.471086] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1935.904079] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 07:25:20 (1787570720) [ 1936.062202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1936.133508] LNet: 94596:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1936.141797] LNetError: 94596:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1936.181422] LNet: Removed LNI 192.168.204.52@tcp [ 1936.217761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1936.279635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1936.316350] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1945.685692] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 07:25:30 (1787570730) [ 1947.638966] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1950.189569] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 07:25:34 (1787570734) [ 1950.944909] LNet: 95636:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1950.951253] LNetError: 95636:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1950.969458] LNet: Removed LNI 192.168.204.52@tcp [ 1951.787667] Key type .llcrypt unregistered [ 1951.789295] Key type ._llcrypt unregistered [ 1959.474939] Key type ._llcrypt registered [ 1959.484432] Key type .llcrypt registered [ 1959.701039] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1959.721909] alg: No test for adler32 (adler32-zlib) [ 1960.662180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1960.692220] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1968.211125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.152@tcp [ 1973.026290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.204.152@tcp [ 1973.614769] LNet: 96061:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1973.634465] LNetError: 96061:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1973.673791] LNet: Removed LNI 192.168.204.52@tcp [ 1974.306718] Key type .llcrypt unregistered [ 1974.311301] Key type ._llcrypt unregistered [ 1988.636629] Key type ._llcrypt registered [ 1988.638988] Key type .llcrypt registered [ 1990.110272] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1991.899401] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1993.780724] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 07:26:18 (1787570778) [ 1995.609752] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1997.955908] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 07:26:22 (1787570782) [ 2000.128795] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg452-server [ 2002.066343] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 07:26:27 (1787570787) [ 2003.914161] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg452-server [ 2006.324574] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 07:26:30 (1787570790) [ 2008.372220] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2008.447672] alg: No test for adler32 (adler32-zlib) [ 2009.662392] Lustre: Lustre: Build Version: 2.17.57_80_ga77f2be [ 2010.210191] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 2012.023791] Key type lgssc registered [ 2013.567448] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2037.783705] Key type lgssc unregistered [ 2038.080119] LNet: 99241:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2038.092142] LNetError: 99241:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2038.131121] LNet: Removed LNI 192.168.204.52@tcp [ 2038.955526] Key type .llcrypt unregistered [ 2038.958122] Key type ._llcrypt unregistered [ 2056.077865] Key type ._llcrypt registered [ 2056.081743] Key type .llcrypt registered [ 2057.635258] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 07:27:22 (1787570842) [ 2058.152222] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2058.162404] alg: No test for adler32 (adler32-zlib) [ 2059.196109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2059.275237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2059.327175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2059.376915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2059.433751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2066.847484] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 07:27:31 (1787570851) [ 2067.115665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2067.164863] LNet: 100655:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2067.226305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2067.325992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2067.366728] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 2068.701248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2068.725914] LNet: 100800:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2068.731922] LNetError: 100800:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2068.747650] LNet: Removed LNI 192.168.204.52@tcp [ 2068.792107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2068.894271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2068.926041] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 2069.640411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2069.690661] LNetError: 100828:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2069.717718] LNet: Removed LNI 192.168.204.52@tcp [ 2069.768505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2069.902644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2069.955300] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 2077.610719] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 07:27:42 (1787570862) [ 2077.922566] LNet: 101419:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2077.932015] LNet: 101419:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2077.952138] LNetError: 101419:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2077.978703] LNet: Removed LNI 192.168.204.52@tcp [ 2078.458597] Key type .llcrypt unregistered [ 2078.460444] Key type ._llcrypt unregistered [ 2086.154781] Key type ._llcrypt registered [ 2086.162411] Key type .llcrypt registered [ 2086.440387] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2086.471334] alg: No test for adler32 (adler32-zlib) [ 2087.442387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2087.574782] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 2087.577423] LNet: *** cfs_fail_loc=e003, val=100*** [ 2088.476886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 2088.541058] LNet: *** cfs_fail_loc=e003, val=100*** [ 2088.542851] LNet: Skipped 2 previous similar messages [ 2088.557798] Ping reply received with mapping: [ 2088.561355] found metadata reply NI[192.168.204.52@tcp] [ 2088.564312] 34 cc a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2088.567898] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2088.571391] 5a 5a 5a 5a 5a 5a 5a 5a [ 2088.818788] LNet: 101820:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2088.826449] LNetError: 101820:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2089.898929] LNet: Removed LNI 192.168.204.52@tcp [ 2090.894549] Key type .llcrypt unregistered [ 2090.900656] Key type ._llcrypt unregistered [ 2104.551763] Key type ._llcrypt registered [ 2104.553463] Key type .llcrypt registered [ 2106.107680] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 07:28:10 (1787570890) [ 2106.784184] Key type .llcrypt unregistered [ 2106.786723] Key type ._llcrypt unregistered [ 2114.314048] Key type ._llcrypt registered [ 2114.316187] Key type .llcrypt registered [ 2114.508254] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2114.522880] alg: No test for adler32 (adler32-zlib) [ 2115.743648] LNetError: 102878:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2116.293407] Key type .llcrypt unregistered [ 2116.299361] Key type ._llcrypt unregistered [ 2122.164116] Key type ._llcrypt registered [ 2122.166378] Key type .llcrypt registered [ 2123.681543] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 07:28:28 (1787570908) [ 2124.247500] Key type .llcrypt unregistered [ 2124.251855] Key type ._llcrypt unregistered [ 2131.389217] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2131.392799] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2133.946301] Key type ._llcrypt registered [ 2133.949044] Key type .llcrypt registered [ 2134.199770] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2134.233314] alg: No test for adler32 (adler32-zlib) [ 2135.251476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2135.354827] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2146.295169] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 07:28:51 (1787570931) [ 2146.691787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2146.708901] LNet: 104682:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2146.721344] LNetError: 104682:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2146.746225] LNet: Removed LNI 10.1.2.3@tcp [ 2146.776898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2147.554521] Key type .llcrypt unregistered [ 2147.557495] Key type ._llcrypt unregistered [ 2154.037929] Key type ._llcrypt registered [ 2154.043705] Key type .llcrypt registered [ 2154.385977] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2154.405971] alg: No test for adler32 (adler32-zlib) [ 2155.455917] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 2155.551474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 2155.721946] LNet: 104974:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2155.732098] LNetError: 104974:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2156.778301] LNet: Removed LNI 192.168.204.52@tcp [ 2157.674492] Key type .llcrypt unregistered [ 2157.680197] Key type ._llcrypt unregistered [ 2170.240951] Key type ._llcrypt registered [ 2170.244307] Key type .llcrypt registered [ 2171.816723] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 07:29:16 (1787570956) [ 2172.716127] Key type .llcrypt unregistered [ 2172.721429] Key type ._llcrypt unregistered [ 2180.412520] Key type ._llcrypt registered [ 2180.415193] Key type .llcrypt registered [ 2180.716168] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2180.739863] alg: No test for adler32 (adler32-zlib) [ 2189.208642] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 07:29:33 (1787570973) [ 2189.386221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2189.430035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2189.528669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2189.557662] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 2190.307732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2190.371787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2190.419500] LNet: Removed LNI 192.168.204.52@tcp [ 2190.869561] LNet: 106730:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2191.304453] Key type .llcrypt unregistered [ 2191.307656] Key type ._llcrypt unregistered [ 2205.974765] Key type ._llcrypt registered [ 2205.985336] Key type .llcrypt registered [ 2208.061345] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 07:29:51 (1787570991) [ 2208.599733] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2208.618350] alg: No test for adler32 (adler32-zlib) [ 2209.529280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2209.593980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2209.627077] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 2210.597276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 2210.635795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.204.52@tcp [ 2210.662731] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2210.667125] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2210.672164] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2210, 0, 0 [ 2210.679506] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2210.686150] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2210.689233] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2210.693192] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2210, 0, 0 [ 2210.699594] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2210.702448] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2210.711930] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2210.715202] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2210, 0, 2260 [ 2210.730757] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2210.735618] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2210.737341] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2210.742152] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2210, 0, 2260 [ 2210.747796] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2218.830427] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 07:30:03 (1787571003) [ 2219.374340] LNet: 108272:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2219.397097] LNetError: 108272:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2219.434199] LNet: Removed LNI 192.168.204.52@tcp [ 2220.179739] Key type .llcrypt unregistered [ 2220.188339] Key type ._llcrypt unregistered [ 2227.216521] Key type ._llcrypt registered [ 2227.221556] Key type .llcrypt registered [ 2227.524579] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2227.556536] alg: No test for adler32 (adler32-zlib) [ 2228.527210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2228.570284] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 2236.523957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.152@tcp [ 2247.313874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2247.391025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2247.408901] LNet: Added LNI 192.168.204.52@tcp1 [8/256/0/180] [ 2248.287824] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2248.558136] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2248.573521] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2248.581063] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2250.957949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2251.003304] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2261.979130] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 07:30:46 (1787571046) [ 2262.168348] LNet: 109782:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2262.180658] LNetError: 109782:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2262.195486] LNet: Removed LNI 192.168.204.52@tcp [ 2263.200656] LNet: Removed LNI 192.168.204.52@tcp1 [ 2263.223856] LNetError: 109782:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2263.230954] LNetError: 109782:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2263.255357] LNet: Removed LNI 10.1.2.3@tcp99 [ 2264.223556] Key type .llcrypt unregistered [ 2264.226915] Key type ._llcrypt unregistered [ 2272.017301] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2272.022221] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2274.607693] Key type ._llcrypt registered [ 2274.613478] Key type .llcrypt registered [ 2274.851316] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2274.869667] alg: No test for adler32 (adler32-zlib) [ 2275.756813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2275.873884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2275.913060] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 2276.802199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2276.819820] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2279.156153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2279.216125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2279.258908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2279.291485] LNetError: 110535:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2279.324182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2279.376969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.52@tcp [ 2280.117493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2280.212453] LNet: 110555:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2280.223859] LNetError: 110555:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2280.232224] LNet: Removed LNI 10.1.2.3@tcp [ 2281.254941] LNet: Removed LNI 192.168.204.52@tcp [ 2282.324457] Key type .llcrypt unregistered [ 2282.328992] Key type ._llcrypt unregistered [ 2297.316679] Key type ._llcrypt registered [ 2297.320661] Key type .llcrypt registered [ 2298.731269] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 07:31:23 (1787571083) [ 2299.147749] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2299.174708] alg: No test for adler32 (adler32-zlib) [ 2300.169336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2300.258380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2300.316751] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 2301.137815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 2301.229358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg452-client@tcp [ 2308.883260] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 07:31:33 (1787571093) [ 2309.051654] LNet: 112085:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2309.060664] LNetError: 112085:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2309.099208] LNet: Removed LNI 192.168.204.52@tcp [ 2310.288639] Key type .llcrypt unregistered [ 2310.292654] Key type ._llcrypt unregistered [ 2318.609995] Key type ._llcrypt registered [ 2318.615528] Key type .llcrypt registered [ 2318.850950] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2318.880298] alg: No test for adler32 (adler32-zlib) [ 2319.948119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2319.997466] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 2320.034760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2327.677554] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 07:31:52 (1787571112) [ 2329.764625] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2331.815884] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 07:31:56 (1787571116) [ 2333.419353] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2335.549653] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 07:32:00 (1787571120) [ 2335.823815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2335.847156] LNet: 113518:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2335.857771] LNetError: 113518:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2335.888830] LNet: Removed LNI 192.168.204.52@tcp [ 2335.922268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2336.022252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2336.051905] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 2336.950649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2336.999567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2344.712976] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 07:32:09 (1787571129) [ 2345.161893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2345.194375] LNet: 114228:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2345.209342] LNetError: 114228:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2345.228955] LNet: Removed LNI 192.168.204.52@tcp [ 2345.274632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2345.335687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2345.388640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2346.035378] Key type .llcrypt unregistered [ 2346.037074] Key type ._llcrypt unregistered [ 2359.787087] Key type ._llcrypt registered [ 2359.789178] Key type .llcrypt registered [ 2361.275672] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 07:32:25 (1787571145) [ 2361.709407] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2361.743522] alg: No test for adler32 (adler32-zlib) [ 2362.837243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2362.902906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2362.927132] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 2363.995263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2364.069446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.52@tcp [ 2372.478131] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 07:32:36 (1787571156) [ 2372.689586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2372.749855] LNet: 115785:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2372.767817] LNetError: 115785:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2372.809210] LNet: Removed LNI 192.168.204.52@tcp [ 2372.853923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2372.908245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2372.941321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2380.294889] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 07:32:45 (1787571165) [ 2380.410649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2380.454545] LNet: 116363:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2380.487785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2380.582705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2381.081766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2381.142985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2381.264648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2381.777076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2381.812072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2381.928206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2381.961758] LNet: 116401:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2381.996539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2382.106563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2382.614167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2382.655459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2382.761671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2383.367479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2383.408561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2390.603771] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 07:32:55 (1787571175) [ 2390.723065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2390.761952] LNet: 116994:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2390.817621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2390.920786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2390.966505] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 2391.860366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2391.887568] LNet: Added LNI 192.168.204.52@tcp2 [8/256/0/180] [ 2392.872480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2392.947572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.52@tcp [ 2393.167308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2393.197346] LNetError: 117281:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2394.271210] LNet: Removed LNI 192.168.204.52@tcp [ 2394.282428] LNetError: 117281:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2394.307192] LNet: Removed LNI 192.168.204.52@tcp2 [ 2394.358425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2394.447234] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 2394.455397] LNet: Added LNI 192.168.204.52@tcp2 [8/256/0/180] [ 2394.461862] LNet: 117287:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2395.110355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.52@tcp [ 2395.330396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2395.349820] LNet: 117324:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2395.356504] LNet: 117324:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2396.383288] LNet: Removed LNI 192.168.204.52@tcp [ 2396.395219] LNetError: 117324:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2396.404936] LNetError: 117324:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2396.423391] LNet: Removed LNI 192.168.204.52@tcp2 [ 2396.477847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2396.567561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2396.594177] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 2397.546351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2397.575961] LNet: Added LNI 192.168.204.52@tcp2 [8/256/0/180] [ 2398.597509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.52@tcp [ 2398.635723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2398.737472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.52@tcp [ 2398.949416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2400.032511] LNet: Removed LNI 192.168.204.52@tcp [ 2401.064491] LNet: Removed LNI 192.168.204.52@tcp2 [ 2401.098569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2401.152403] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 2401.161343] LNet: Added LNI 192.168.204.52@tcp2 [8/256/0/180] [ 2401.170233] LNet: 117615:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2401.840372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.52@tcp [ 2402.028652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2402.051287] LNetError: 117652:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2402.058120] LNetError: 117652:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2402.072436] LNet: Removed LNI 192.168.204.52@tcp [ 2403.112514] LNet: Removed LNI 192.168.204.52@tcp2 [ 2403.147642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2403.228525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2403.273458] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 2403.916522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2403.943224] LNet: Added LNI 192.168.204.52@tcp2 [8/256/0/180] [ 2404.977956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2405.043740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.52@tcp [ 2405.274983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2405.308957] LNet: 117932:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2405.317106] LNet: 117932:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2406.367511] LNet: Removed LNI 192.168.204.52@tcp [ 2406.382546] LNet: Removed LNI 192.168.204.52@tcp2 [ 2406.428189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2406.550655] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 2406.558635] LNet: Added LNI 192.168.204.52@tcp2 [8/256/0/180] [ 2406.566344] LNet: 117938:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2407.035766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.52@tcp [ 2407.313885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2408.415448] LNet: Removed LNI 192.168.204.52@tcp [ 2408.431685] LNet: Removed LNI 192.168.204.52@tcp2 [ 2408.456534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2408.519658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2408.549635] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 2409.269268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2409.290941] LNet: Added LNI 192.168.204.52@tcp2 [8/256/0/180] [ 2410.342888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.52@tcp [ 2410.385385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2410.462457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.52@tcp [ 2410.753389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2410.781961] LNetError: 118260:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2410.786608] LNetError: 118260:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2411.809492] LNet: Removed LNI 192.168.204.52@tcp [ 2412.847154] LNet: Removed LNI 192.168.204.52@tcp2 [ 2412.920550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2413.021449] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 2413.037480] LNet: Added LNI 192.168.204.52@tcp2 [8/256/0/180] [ 2413.042942] LNet: 118266:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2413.676294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.52@tcp [ 2413.936025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2413.959252] LNet: Removed LNI 192.168.204.52@tcp [ 2415.018354] LNet: Removed LNI 192.168.204.52@tcp2 [ 2415.064281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2415.143732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2415.194965] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 2416.002397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2416.027903] LNet: Added LNI 192.168.204.52@tcp2 [8/256/0/180] [ 2416.760798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2416.786777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2416.847647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.52@tcp [ 2417.161379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2418.207205] LNet: Removed LNI 192.168.204.52@tcp [ 2418.221370] LNet: Removed LNI 192.168.204.52@tcp2 [ 2418.284770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2418.373284] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 2418.386367] LNet: Added LNI 192.168.204.52@tcp2 [8/256/0/180] [ 2418.393235] LNet: 118606:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2419.043545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.52@tcp [ 2419.398260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2420.447292] LNet: Removed LNI 192.168.204.52@tcp [ 2420.457967] LNet: Removed LNI 192.168.204.52@tcp2 [ 2420.492181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2420.611160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2420.651911] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 2421.521209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2421.546189] LNet: Added LNI 192.168.204.52@tcp2 [8/256/0/180] [ 2422.468800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.52@tcp [ 2422.517585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2422.545948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2422.614973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.52@tcp [ 2422.984363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2423.007744] LNet: 118960:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2423.012476] LNet: 118960:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 2424.031320] LNet: Removed LNI 192.168.204.52@tcp [ 2425.068186] LNet: Removed LNI 192.168.204.52@tcp2 [ 2425.096365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2425.176622] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 2425.181729] LNet: Added LNI 192.168.204.52@tcp2 [8/256/0/180] [ 2425.926995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.52@tcp [ 2433.547747] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2435.054289] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 07:33:40 (1787571220) [ 2435.277149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2435.310576] LNetError: 119726:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2435.322992] LNetError: 119726:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 9 previous similar messages [ 2435.338068] LNet: Removed LNI 192.168.204.52@tcp [ 2435.359343] LNet: Removed LNI 192.168.204.52@tcp2 [ 2435.395253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2435.686842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2435.710679] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 2436.428573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2436.454943] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2437.273671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2437.300093] LNet: Removed LNI 10.1.2.3@tcp [ 2437.315153] LNet: Removed LNI 192.168.204.52@tcp [ 2437.373091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2437.466625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2437.491795] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 2437.496201] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2438.099456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2438.126829] LNet: Removed LNI 192.168.204.52@tcp [ 2438.146444] LNet: Removed LNI 10.1.2.3@tcp [ 2445.479629] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 07:33:50 (1787571230) [ 2445.756342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2445.796410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2445.864631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2445.892140] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 2446.730357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2446.769750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 2446.808481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 2446.851780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 2446.879749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 2446.907317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 2446.937436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 2446.962174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 2446.989649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 2447.017481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 2447.048996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 2447.199377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2453.810636] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 07:33:58 (1787571238) [ 2454.309314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2455.398699] LNet: Removed LNI 192.168.204.52@tcp [ 2455.449187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2455.575809] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2455.713429] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2455.718149] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2455.723830] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2458.102388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2458.124770] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 2458.862453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2458.911607] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2461.145522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.52@tcp [ 2461.205223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2461.308550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.204.52@tcp -m PUT -e local_timeout -r 1 [ 2461.430484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2461.528560] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2461.540264] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2461.814691] LNetError: 121407:0:(lib-move.c:4801:lnet_parse()) 192.168.204.52@tcp, src 192.168.204.52@tcp: Dropping PUT (error -108 looking up sender) [ 2462.560758] LNet: 121870:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2462.567454] LNet: 121870:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2462.595494] LNet: Removed LNI 10.1.2.3@tcp [ 2463.661853] LNet: Removed LNI 192.168.204.52@tcp [ 2464.136633] Key type .llcrypt unregistered [ 2464.142174] Key type ._llcrypt unregistered [ 2476.186164] Key type ._llcrypt registered [ 2476.187820] Key type .llcrypt registered [ 2477.269833] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 07:34:22 (1787571262) [ 2477.802181] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2477.830646] alg: No test for adler32 (adler32-zlib) [ 2478.787645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2479.132656] LNet: 122723:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2479.666376] Key type .llcrypt unregistered [ 2479.669101] Key type ._llcrypt unregistered [ 2484.802993] Key type ._llcrypt registered [ 2484.808942] Key type .llcrypt registered [ 2485.027606] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2485.049771] alg: No test for adler32 (adler32-zlib) [ 2485.965399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2486.001880] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 2493.755343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.152@tcp [ 2493.970458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2494.662437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.152@tcp [ 2494.883865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2494.906930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2494.938912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2494.968700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2494.993578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2495.014586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2495.039341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2495.061707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2495.075651] LNet: 123177:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2495.090745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2495.141427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2495.191691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2495.225640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2495.263241] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.152@tcp [ 2495.274571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2495.312388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2495.921684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2496.140986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2496.171172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2496.215697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2496.248495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2496.294061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2496.335979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2496.363761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2496.401685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2496.419334] LNet: 123294:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2496.444407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2496.495584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2496.546630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2496.602405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2496.631740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2496.677299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2497.282111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.152@tcp [ 2497.452140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2497.472530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2497.496474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2497.520701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2497.545333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2497.583280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2497.626707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2497.652806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2497.667532] LNet: 123411:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2497.701030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2497.722773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2497.771521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2497.800189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2497.826930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2497.856307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2498.606935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2498.807579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2498.833306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2498.856429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2498.887480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2498.923288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2498.951180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2498.985756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2499.015327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2499.037823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2499.061753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2499.083522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2499.108336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2499.130832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2499.156969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2499.557296] LNet: 123579:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2499.580621] LNetError: 123579:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2499.603639] LNet: Removed LNI 192.168.204.52@tcp [ 2500.248992] Key type .llcrypt unregistered [ 2500.257060] Key type ._llcrypt unregistered [ 2516.142336] Key type ._llcrypt registered [ 2516.146353] Key type .llcrypt registered [ 2517.488903] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 07:35:02 (1787571302) [ 2518.086616] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2518.105986] alg: No test for adler32 (adler32-zlib) [ 2519.057852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2519.624977] LNet: 124485:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2520.325748] Key type .llcrypt unregistered [ 2520.338104] Key type ._llcrypt unregistered [ 2526.425529] Key type ._llcrypt registered [ 2526.431276] Key type .llcrypt registered [ 2526.710580] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2526.725878] alg: No test for adler32 (adler32-zlib) [ 2527.587588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2527.626220] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 2535.761313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.152@tcp [ 2545.299043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2545.363626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2545.401795] LNet: Added LNI 192.168.204.52@tcp1 [8/256/0/180] [ 2546.319254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2546.890387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.152@tcp [ 2547.103443] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.152@tcp [ 2547.141368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2547.171921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2547.202929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2547.243600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2547.286258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2547.323530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2547.368964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2547.426195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2547.462545] LNet: 125142:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2547.489695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2547.523674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2547.573908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2547.622529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2547.660726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2547.707179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2548.267950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2548.476538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2548.500907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2548.527227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2548.557627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2548.585438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2548.614987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2548.650437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2548.671881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2548.686672] LNet: 125259:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2548.703983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2548.736255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2548.768644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2548.794508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2548.825501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2548.859419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2549.529462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.152@tcp [ 2549.730111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2549.763065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2549.810124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2549.840459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2549.861364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2549.888757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2549.921439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2549.946532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2549.960938] LNet: 125377:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2549.978526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2550.010875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2550.043441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2550.070540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2550.095691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2550.137468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2550.753354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2550.972459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2551.003409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2551.034656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2551.064964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2551.092703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2551.134556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2551.163755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2551.210115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2551.240608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2551.280538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2551.335992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2551.364827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2551.396803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2551.427662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2556.432449] LNet: 125579:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2556.438506] LNetError: 125579:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2556.445092] LNet: Removed LNI 192.168.204.52@tcp [ 2556.456689] LNet: Removed LNI 192.168.204.52@tcp1 [ 2556.913139] Key type .llcrypt unregistered [ 2556.924554] Key type ._llcrypt unregistered [ 2573.331832] Key type ._llcrypt registered [ 2573.341179] Key type .llcrypt registered [ 2574.519539] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 07:35:59 (1787571359) [ 2575.054956] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2575.076767] alg: No test for adler32 (adler32-zlib) [ 2575.947725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2575.990732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2576.010484] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 2576.511343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2576.534263] LNet: Added LNI 192.168.204.52@tcp2 [8/256/0/180] [ 2577.276636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.52@tcp [ 2577.346134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2578.399223] LNet: Removed LNI 192.168.204.52@tcp [ 2583.384561] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 07:36:08 (1787571368) [ 2584.624988] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2586.578928] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 07:36:11 (1787571371) [ 2593.466197] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 07:36:18 (1787571378) [ 2600.187895] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 07:36:25 (1787571385) [ 2608.027214] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 07:36:32 (1787571392) [ 2615.004453] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 07:36:39 (1787571399) [ 2622.287468] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 07:36:47 (1787571407) [ 2629.548824] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 07:36:54 (1787571414) [ 2636.816907] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 07:37:02 (1787571422) [ 2645.845340] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 07:37:10 (1787571430) [ 2653.945907] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 07:37:18 (1787571438) [ 2661.083560] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 07:37:25 (1787571445) [ 2668.355273] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 07:37:33 (1787571453) [ 2675.133088] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 07:37:39 (1787571459) [ 2681.458306] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 07:37:46 (1787571466) [ 2688.923399] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 07:37:53 (1787571473) [ 2698.026718] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 07:38:02 (1787571482) [ 2706.230765] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 07:38:11 (1787571491) [ 2714.093701] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 07:38:18 (1787571498) [ 2714.457605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2714.493431] LNet: 137272:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2714.499600] LNetError: 137272:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2714.534624] LNet: Removed LNI 192.168.204.52@tcp2 [ 2714.557861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2714.696503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2714.724904] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 2716.068192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2716.105628] LNet: 137414:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2716.111369] LNetError: 137414:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2716.129718] LNet: Removed LNI 192.168.204.52@tcp [ 2716.154569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2716.225426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2716.249722] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 2723.936694] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 07:38:28 (1787571508) [ 2733.241830] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 07:38:37 (1787571517) [ 2745.282229] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 07:38:49 (1787571529) [ 2754.655766] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 07:38:58 (1787571538) [ 2763.881815] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 07:39:08 (1787571548) [ 2772.826542] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 07:39:17 (1787571557) [ 2782.190833] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 07:39:26 (1787571566) [ 2791.499306] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 07:39:36 (1787571576) [ 2801.315183] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 07:39:45 (1787571585) [ 2809.968664] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 07:39:54 (1787571594) [ 2818.476279] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 07:40:02 (1787571602) [ 2827.187160] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 07:40:11 (1787571611) [ 2836.513954] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2606 sec ========== 07:40:20 (1787571620) [ 2837.012792] LNet: 144885:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2837.020196] LNetError: 144885:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2837.042723] LNet: Removed LNI 192.168.204.52@tcp [ 2838.017940] Key type .llcrypt unregistered [ 2838.026035] Key type ._llcrypt unregistered [ 2851.559811] Key type ._llcrypt registered [ 2851.578652] Key type .llcrypt registered [ 2852.216203] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2852.240836] alg: No test for adler32 (adler32-zlib) [ 2853.383269] Lustre: Lustre: Build Version: 2.17.57_80_ga77f2be [ 2853.622314] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 2855.327205] Key type lgssc registered [ 2856.734311] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2931.923097] Lustre: Mounted lustre-client - version 2.17.57_80_ga77f2be [ 2937.523192] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2945.863979] Lustre: Unmounted lustre-client [ 2983.023950] Key type lgssc unregistered [ 2983.335207] LNet: 147846:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2983.347327] LNetError: 147846:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2983.379325] LNet: Removed LNI 192.168.204.52@tcp [ 2984.245298] Key type .llcrypt unregistered [ 2984.251950] Key type ._llcrypt unregistered