[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 427520132 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002386] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004016] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008012] pid_max: default: 32768 minimum: 301 [ 0.009155] LSM: Security Framework initializing [ 0.010094] Yama: becoming mindful. [ 0.011041] SELinux: Initializing. [ 0.012076] *** VALIDATE selinux *** [ 0.019701] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024484] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025133] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026116] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027109] *** VALIDATE tmpfs *** [ 0.029424] *** VALIDATE proc *** [ 0.031112] *** VALIDATE cgroup *** [ 0.032009] *** VALIDATE cgroup2 *** [ 0.033300] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034189] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036041] Spectre V2 : User space: Vulnerable [ 0.037011] Speculative Store Bypass: Vulnerable [ 0.040237] debug: unmapping init [mem 0xffffffffb0859000-0xffffffffb0860fff] [ 0.042131] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043639] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044027] ... version: 2 [ 0.045011] ... bit width: 48 [ 0.046014] ... generic registers: 4 [ 0.047013] ... value mask: 0000ffffffffffff [ 0.048014] ... max period: 00007fffffffffff [ 0.049015] ... fixed-purpose events: 3 [ 0.050013] ... event mask: 000000070000000f [ 0.051279] rcu: Hierarchical SRCU implementation. [ 0.053489] smp: Bringing up secondary CPUs ... [ 0.054577] x86: Booting SMP configuration: [ 0.055035] .... node #0, CPUs: #1 #2 #3 [ 0.059016] smp: Brought up 1 node, 4 CPUs [ 0.061014] smpboot: Max logical packages: 1 [ 0.062015] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.137051] node 0 deferred pages initialised in 71ms [ 0.141012] devtmpfs: initialized [ 0.142319] x86/mm: Memory block size: 128MB [ 0.145842] gcov: version magic: 0x41383552 [ 0.148309] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.152107] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.154280] pinctrl core: initialized pinctrl subsystem [ 0.156193] [ 0.156897] ************************************************************* [ 0.158011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.160010] ** ** [ 0.162016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.164016] ** ** [ 0.167012] ** This means that this kernel is built to expose internal ** [ 0.169011] ** IOMMU data structures, which may compromise security on ** [ 0.173016] ** your system. ** [ 0.174011] ** ** [ 0.176011] ** If you see this message and you are not debugging the ** [ 0.178013] ** kernel, report this immediately to your vendor! ** [ 0.180015] ** ** [ 0.183014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.186014] ************************************************************* [ 0.188604] NET: Registered protocol family 16 [ 0.191446] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.193050] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.196060] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.199104] cpuidle: using governor menu [ 0.200974] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.204466] PCI: Using configuration type 1 for base access [ 0.206125] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.216151] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.219018] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.223161] cryptd: max_cpu_qlen set to 1000 [ 0.227105] ACPI: Added _OSI(Module Device) [ 0.228018] ACPI: Added _OSI(Processor Device) [ 0.229013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.230011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.233666] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.240649] ACPI: Interpreter enabled [ 0.242095] ACPI: PM: (supports S0 S3 S4 S5) [ 0.244039] ACPI: Using IOAPIC for interrupt routing [ 0.245191] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.249481] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.262814] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.266067] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.270030] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.274168] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.281128] acpiphp: Slot [2] registered [ 0.284139] acpiphp: Slot [5] registered [ 0.286299] acpiphp: Slot [6] registered [ 0.287207] acpiphp: Slot [3] registered [ 0.289128] acpiphp: Slot [4] registered [ 0.290287] acpiphp: Slot [7] registered [ 0.292138] acpiphp: Slot [8] registered [ 0.294125] acpiphp: Slot [9] registered [ 0.296108] acpiphp: Slot [10] registered [ 0.297108] acpiphp: Slot [11] registered [ 0.299109] acpiphp: Slot [12] registered [ 0.301117] acpiphp: Slot [13] registered [ 0.302156] acpiphp: Slot [14] registered [ 0.304122] acpiphp: Slot [15] registered [ 0.305094] acpiphp: Slot [16] registered [ 0.306102] acpiphp: Slot [17] registered [ 0.308104] acpiphp: Slot [18] registered [ 0.310122] acpiphp: Slot [19] registered [ 0.312119] acpiphp: Slot [20] registered [ 0.313115] acpiphp: Slot [21] registered [ 0.315126] acpiphp: Slot [22] registered [ 0.317123] acpiphp: Slot [23] registered [ 0.319161] acpiphp: Slot [24] registered [ 0.320137] acpiphp: Slot [25] registered [ 0.322275] acpiphp: Slot [26] registered [ 0.324228] acpiphp: Slot [27] registered [ 0.326278] acpiphp: Slot [28] registered [ 0.328179] acpiphp: Slot [29] registered [ 0.330170] acpiphp: Slot [30] registered [ 0.332163] acpiphp: Slot [31] registered [ 0.333084] PCI host bridge to bus 0000:00 [ 0.335028] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.338034] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.340044] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.344037] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.347041] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.350045] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.352186] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.355111] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.358320] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.367015] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.371064] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.375021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.377017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.380024] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.383647] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.386862] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.389045] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.391856] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.396015] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.405016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.410016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.416237] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.424021] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.431023] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.451026] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.462917] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.470018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.480025] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.492026] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.504018] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.507716] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.510424] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.512427] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.515278] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.522138] iommu: Default domain type: Passthrough [ 0.523613] SCSI subsystem initialized [ 0.524176] ACPI: bus type USB registered [ 0.525187] usbcore: registered new interface driver usbfs [ 0.527122] usbcore: registered new interface driver hub [ 0.530118] usbcore: registered new device driver usb [ 0.532202] pps_core: LinuxPPS API ver. 1 registered [ 0.534015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.538084] PTP clock support registered [ 0.541158] EDAC MC: Ver: 3.0.0 [ 0.542325] PCI: Using ACPI for IRQ routing [ 0.544915] NetLabel: Initializing [ 0.546013] NetLabel: domain hash size = 128 [ 0.548013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.549102] NetLabel: unlabeled traffic allowed by default [ 0.552363] vgaarb: loaded [ 0.553364] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.556019] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.563166] clocksource: Switched to clocksource kvm-clock [ 0.676726] VFS: Disk quotas dquot_6.6.0 [ 0.678394] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.680757] *** VALIDATE ramfs *** [ 0.682112] *** VALIDATE hugetlbfs *** [ 0.683877] pnp: PnP ACPI init [ 0.686343] pnp: PnP ACPI: found 6 devices [ 0.712669] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.716765] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.719225] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.721716] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.724436] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.727032] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.730223] NET: Registered protocol family 2 [ 0.733072] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.738749] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.742615] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.748807] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.753130] TCP: Hash tables configured (established 65536 bind 65536) [ 0.756513] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.760496] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.763781] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.766940] NET: Registered protocol family 1 [ 0.770073] RPC: Registered named UNIX socket transport module. [ 0.772707] RPC: Registered udp transport module. [ 0.774517] RPC: Registered tcp transport module. [ 0.776492] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.779306] NET: Registered protocol family 44 [ 0.781100] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.783607] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.786013] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.788471] PCI: CLS 0 bytes, default 64 [ 0.790335] Unpacking initramfs... [ 2.225464] debug: unmapping init [mem 0xffff98bdfcc64000-0xffff98bdfffcffff] [ 2.229791] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.231601] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.233869] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.750215] Initialise system trusted keyrings [ 2.751684] Key type blacklist registered [ 2.756836] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.765695] zbud: loaded [ 2.769222] *** VALIDATE nfs *** [ 2.770604] *** VALIDATE nfs4 *** [ 2.772482] pstore: using deflate compression [ 2.775863] Platform Keyring initialized [ 2.882925] NET: Registered protocol family 38 [ 2.885665] Key type asymmetric registered [ 2.887826] Asymmetric key parser 'x509' registered [ 2.889763] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.893269] io scheduler mq-deadline registered [ 2.895009] io scheduler kyber registered [ 2.896656] io scheduler bfq registered [ 2.899087] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.902336] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.905383] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.908167] ACPI: Power Button [PWRF] [ 2.913197] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.919755] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.930447] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.958152] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.988202] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.998971] Non-volatile memory driver v1.3 [ 3.001139] Linux agpgart interface v0.103 [ 3.037486] virtio_blk virtio1: [vda] 145920 512-byte logical blocks (74.7 MB/71.3 MiB) [ 3.040697] vda: detected capacity change from 0 to 74711040 [ 3.054885] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.057966] vdb: detected capacity change from 0 to 1073741824 [ 3.064289] libphy: Fixed MDIO Bus: probed [ 3.068839] usbcore: registered new interface driver usbserial_generic [ 3.070799] usbserial: USB Serial support registered for generic [ 3.072500] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.075679] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.077567] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.081261] mousedev: PS/2 mouse device common for all mice [ 3.084520] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.088328] rtc_cmos 00:05: RTC can wake from S4 [ 3.093179] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.094816] rtc_cmos 00:05: registered as rtc0 [ 3.099064] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.100418] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.107803] intel_pstate: CPU model not supported [ 3.111465] hid: raw HID events driver (C) Jiri Kosina [ 3.113978] usbcore: registered new interface driver usbhid [ 3.116147] usbhid: USB HID core driver [ 3.118272] drop_monitor: Initializing network drop monitor service [ 3.121319] Initializing XFRM netlink socket [ 3.123677] NET: Registered protocol family 10 [ 3.127416] Segment Routing with IPv6 [ 3.128894] NET: Registered protocol family 17 [ 3.132140] mpls_gso: MPLS GSO support [ 3.138606] RAS: Correctable Errors collector initialized. [ 3.140981] AVX version of gcm_enc/dec engaged. [ 3.142576] AES CTR mode by8 optimization enabled [ 3.222470] sched_clock: Marking stable (3222438540, 0)->(4134045002, -911606462) [ 3.225670] registered taskstats version 1 [ 3.227791] Loading compiled-in X.509 certificates [ 3.229739] zswap: loaded using pool lzo/zbud [ 3.253810] Key type big_key registered [ 3.267814] Key type encrypted registered [ 3.270052] ima: No TPM chip found, activating TPM-bypass! [ 3.272529] ima: Allocated hash algorithm: sha1 [ 3.274524] ima: No architecture policies found [ 3.276426] evm: Initialising EVM extended attributes: [ 3.278470] evm: security.selinux [ 3.279828] evm: security.ima [ 3.281094] evm: security.capability [ 3.282572] evm: HMAC attrs: 0x1 [ 3.285177] rtc_cmos 00:05: setting system clock to 2026-08-14 22:22:31 UTC (1786746151) [ 3.293226] debug: unmapping init [mem 0xffffffffb1803000-0xffffffffb19fffff] [ 3.297155] debug: unmapping init [mem 0xffffffffb0582000-0xffffffffb0858fff] [ 3.304409] Write protecting the kernel read-only data: 28672k [ 3.308031] debug: unmapping init [mem 0xffffffffaec03000-0xffffffffaedfffff] [ 3.311379] debug: unmapping init [mem 0xffffffffaf514000-0xffffffffaf5fffff] [ 3.364230] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.373938] systemd[1]: Detected virtualization kvm. [ 3.375695] systemd[1]: Detected architecture x86-64. [ 3.377372] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.402168] systemd[1]: No hostname configured. [ 3.403422] systemd[1]: Set hostname to . [ 3.405269] random: systemd: uninitialized urandom read (16 bytes read) [ 3.407277] systemd[1]: Initializing machine ID from random generator. [ 3.440778] random: ln: uninitialized urandom read (6 bytes read) [ 3.551761] random: systemd: uninitialized urandom read (16 bytes read) [ 3.554923] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.560329] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.568657] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Reached target Sockets. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Reached target Paths. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Timers. [ OK ] Reached target Swap. Starting Journal Service... Starting Create Volatile Files and Directories... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.234044] device-mapper: uevent: version 1.0.3 [ 4.236837] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon.[ 4.877577] random: fast init done Starting dracut initqueue hook... [ 4.904412] virtio_net virtio0 ens2: renamed from eth0 [ 5.016809] scsi host0: ata_piix [ 5.072808] scsi host1: ata_piix [ 5.080598] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.083595] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.324283] dracut-initqueue[579]: RTNETLINK answers: File exists [ 9.787085] random: crng init done [ 9.788788] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.051983] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.121110] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.361805] SELinux: Disabled at runtime. [ 11.418520] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.426703] systemd[1]: Detected virtualization kvm. [ 11.428617] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.941400] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.945507] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.953935] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.959771] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.963499] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.971315] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.978241] systemd[1]: Created slice system-getty.slice. [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Switch Root. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Remount Root and Kernel File Systems... [ OK ] Reached target rpc_pipefs.target. Mounting Huge Pages File System... [ OK ] Created slice User and Session Slice. [ OK ] Stopped target Initrd File Systems. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Create list of required st…ce nodes for the current kernel... Mounting Kernel Debug File System... [ OK ] Reached target Paths. [ OK ] Reached target Slices. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Mounting POSIX Message Queue File System... Starting Apply Kernel Variables... [ OK ] Stopped target Initrd Root File System. [ 12.108245] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 12.377254] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.651634] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.675529] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.759954] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.776455] EDAC sbridge: Ver: 1.1.2 [ 14.033094] Key type dns_resolver registered [ 14.331081] NFS: Registering the id_resolver key type [ 14.332887] Key type id_resolver registered [ 14.334591] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg622-client login: [ 40.968509] libcfs: loading out-of-tree module taints kernel. [ 41.051549] Key type ._llcrypt registered [ 41.053649] Key type .llcrypt registered [ 41.423742] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 41.432193] alg: No test for adler32 (adler32-zlib) [ 42.476241] Lustre: Lustre: Build Version: 2.17.57_3_g6085841 [ 42.869078] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 44.504164] Key type lgssc registered [ 46.525692] Lustre: Echo OBD driver; http://www.lustre.org/ [ 92.529092] hrtimer: interrupt took 3071803 ns [ 175.589033] Lustre: Mounted lustre-client - version 2.17.57_3_g6085841 [ 180.409873] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 197.302448] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing check_logdir /tmp/testlogs/ [ 201.185086] Lustre: lustre-OST0000-osc-ffff98be42c82000: disconnect after 23s idle [ 202.216431] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing yml_node [ 206.426602] Lustre: DEBUG MARKER: Client: 2.17.57.3 [ 209.255352] Lustre: DEBUG MARKER: MDS: 2.17.57.3 [ 212.089849] Lustre: DEBUG MARKER: OSS: 2.17.57.3 [ 214.280552] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Fri Aug 14 18:26:00 EDT 2026 [ 229.993867] Lustre: Unmounted lustre-client [ 257.768776] Key type lgssc unregistered [ 258.067865] LNet: 8434:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 258.078725] LNetError: 8434:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 258.097225] LNet: Removed LNI 192.168.206.22@tcp [ 258.963144] Key type .llcrypt unregistered [ 258.968499] Key type ._llcrypt unregistered [ 284.676556] Key type ._llcrypt registered [ 284.680947] Key type .llcrypt registered [ 284.901581] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 284.925428] alg: No test for adler32 (adler32-zlib) [ 285.874783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 285.971373] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 286.025571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 289.437586] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 289.683030] LNet: 9321:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 289.694501] LNetError: 9321:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 289.729213] LNet: Removed LNI 192.168.206.22@tcp [ 290.257124] Key type .llcrypt unregistered [ 290.261878] Key type ._llcrypt unregistered [ 296.936067] Key type ._llcrypt registered [ 296.939606] Key type .llcrypt registered [ 298.827092] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 18:27:24 (1786746444) [ 299.218798] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 299.247898] alg: No test for adler32 (adler32-zlib) [ 300.193831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 300.281972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 300.306178] LNet: 9736:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 300.346938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 300.506225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 300.569932] LNet: 9743:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 308.102622] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 18:27:34 (1786746454) [ 308.332795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 308.349694] LNet: 10320:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 308.368081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 308.472983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 315.546802] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 18:27:42 (1786746462) [ 315.701816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 315.731846] LNet: 10902:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 315.764837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 315.874774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 322.756553] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 18:27:49 (1786746469) [ 322.888794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 322.915059] LNet: 11483:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 322.970492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 323.084498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 330.826492] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 18:27:57 (1786746477) [ 330.964737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 330.996707] LNet: 12070:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 331.033748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 331.170026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 331.742525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 337.944400] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 18:28:04 (1786746484) [ 338.026636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 338.071498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 338.188693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 338.661255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 339.094613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 345.518923] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 18:28:12 (1786746492) [ 345.779705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 345.800947] LNet: 13262:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 345.809556] LNet: 13262:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 345.833896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 345.892598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 352.058653] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 18:28:18 (1786746498) [ 352.147580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 352.197986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 352.305799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 352.337992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 352.698559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 352.747494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 353.157034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 353.212746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 353.635603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 353.680585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 354.105596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 354.142972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 354.573834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 354.610886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 354.951099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 354.978760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 355.286404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 355.343410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 355.658328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 355.699099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 356.193703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 356.239469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 356.635340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 356.677644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 357.164728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 357.227497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 357.641801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 357.689394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 364.422293] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 18:28:31 (1786746511) [ 364.508479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 364.539493] LNet: 14522:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 364.544716] LNet: 14522:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 364.568844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 364.694743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 364.749732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 370.859185] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 18:28:37 (1786746517) [ 371.072308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 371.128555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 371.234886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 371.315130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 377.549146] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 18:28:44 (1786746524) [ 377.853304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 377.919781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 378.013951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 378.068094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 385.524434] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 18:28:52 (1786746532) [ 385.940407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 385.992505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 386.101857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 386.163802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 393.185351] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 18:28:59 (1786746539) [ 393.352702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 393.430646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 393.516302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 393.582091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 399.853594] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 18:29:06 (1786746546) [ 399.944845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 399.967134] LNet: 17456:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 399.973149] LNet: 17456:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 400.003196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 400.129046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 400.170784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 406.329958] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 18:29:13 (1786746553) [ 406.397487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 406.432237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 406.540306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 406.827366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 413.538798] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 18:29:20 (1786746560) [ 413.673320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 413.757532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 413.995637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 414.290507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 420.769334] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 18:29:27 (1786746567) [ 420.981794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 421.036601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 421.168375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 421.375834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 427.982419] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 18:29:34 (1786746574) [ 428.087169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 428.139470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 428.730712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 429.035121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 434.944253] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 18:29:41 (1786746581) [ 435.055734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 435.106761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 435.246429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 435.301927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 441.591100] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 18:29:48 (1786746588) [ 441.851681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 441.904751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 441.975539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 442.059691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 448.947504] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 18:29:55 (1786746595) [ 449.051040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 449.104276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 449.203572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 456.048472] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 18:30:02 (1786746602) [ 456.150313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 456.214641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 456.349202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 463.650174] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 18:30:10 (1786746610) [ 463.840554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 463.874407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 463.998344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 464.597974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 470.584491] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 18:30:17 (1786746617) [ 470.722667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 470.755483] LNet: 23484:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 470.764082] LNet: 23484:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 470.796564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 470.897851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 470.931843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 477.219481] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 18:30:24 (1786746624) [ 477.409451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 477.475848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 477.570164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 477.590285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 484.196946] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 18:30:30 (1786746630) [ 484.309539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 484.361759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 484.445410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 484.494238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 491.812393] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 18:30:38 (1786746638) [ 492.299621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 492.355415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 492.450972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 492.498818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 499.964646] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 18:30:46 (1786746646) [ 500.170109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 500.221535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 500.348901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 500.507984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 500.545878] LNetError: 25844:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 500.639397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 500.721364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 507.452110] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 18:30:54 (1786746654) [ 507.728439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 507.752242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 507.812516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.nwefqpTDyZ [ 507.868781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.nwefqpTDyZ [ 507.912457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.nwefqpTDyZ [ 514.666922] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 18:31:01 (1786746661) [ 514.836041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 514.880814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 514.978114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 515.035245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 515.098732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 521.102219] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 18:31:07 (1786746667) [ 521.244874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 521.300217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 521.415313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 521.446669] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 522.867096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.22@tcp [ 523.874092] Lustre: Lustre: Build Version: 2.17.57_3_g6085841 [ 525.648156] Key type lgssc registered [ 526.843182] Lustre: Echo OBD driver; http://www.lustre.org/ [ 530.345823] LNetError: 27605:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.206.22@tcp99 has no available nets [ 533.139484] Key type lgssc unregistered [ 533.502922] LNetError: 28716:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 533.520792] LNet: Removed LNI 192.168.206.22@tcp [ 534.130579] Key type .llcrypt unregistered [ 534.133405] Key type ._llcrypt unregistered [ 540.235809] Key type ._llcrypt registered [ 540.237956] Key type .llcrypt registered [ 541.546443] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 18:31:28 (1786746688) [ 541.886618] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 541.907569] alg: No test for adler32 (adler32-zlib) [ 542.935950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 543.029863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 545.491856] LNet: Message forwarding will be enabled in 2s [ 548.576165] LNet: Message forwarding enabled [ 548.948624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 548.976219] LNet: Message forwarding will stop in 2s to 6s [ 553.502531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 554.100463] LNet: Message forwarding disabled [ 555.795587] LNet: Message forwarding will be enabled in 2s [ 559.273659] LNet: Message forwarding enabled [ 562.470936] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 18:31:49 (1786746709) [ 562.590547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 562.621404] LNet: 30117:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 563.768497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 563.838929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 569.996331] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 18:31:56 (1786746716) [ 570.130234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 570.159961] LNet: 30704:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 570.205777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 570.257778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 570.290502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 570.323906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 570.360755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 570.391367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 570.427609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 570.481788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 570.556333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 570.599175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 570.638271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 570.674907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 570.717828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 570.763434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 570.805489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 570.832943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 570.879734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 570.916877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 570.953435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 570.995450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 571.021341] LNetError: 30749:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 577.071335] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 18:32:03 (1786746723) [ 577.175042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 577.195393] LNet: 31314:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 577.216494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 577.314618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 582.939482] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 18:32:10 (1786746730) [ 584.362884] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg622-server [ 585.849669] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 18:32:13 (1786746733) [ 585.932568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 585.950797] LNet: 32244:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 585.971590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 586.051544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 586.070921] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 586.846465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.206.[23-27]@tcp [ 586.866933] LNet: 32383:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 586.899486] LNetError: 32254:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.23@tcp is being used as a gateway but routing feature is not turned on [ 586.914645] LNetError: 32254:0:(lib-lnet.h:1279:lnet_set_route_aliveness()) route to tcp1 through 192.168.206.23@tcp has gone from up to down [ 587.941048] LNetError: 32254:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.24@tcp is being used as a gateway but routing feature is not turned on [ 587.952546] LNetError: 32254:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 588.963937] LNetError: 32254:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.23@tcp is being used as a gateway but routing feature is not turned on [ 588.975842] LNetError: 32254:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 591.019310] LNetError: 32254:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.23@tcp is being used as a gateway but routing feature is not turned on [ 591.026944] LNetError: 32254:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 592.241710] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 18:32:19 (1786746739) [ 593.753838] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg622-server [ 595.109803] LNetError: 32254:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.23@tcp is being used as a gateway but routing feature is not turned on [ 595.132222] LNetError: 32254:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 595.259384] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 18:32:22 (1786746742) [ 595.401478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 595.442334] LNet: 33315:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 595.456738] LNetError: 33315:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 595.479530] LNet: Removed LNI 192.168.206.22@tcp [ 595.516164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 595.561870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 595.586758] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 596.270402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.206.[23-28/2]@tcp [ 596.305320] LNet: 33436:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 596.321818] LNet: 33436:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 596.360479] LNetError: 33322:0:(lib-lnet.h:1279:lnet_set_route_aliveness()) route to tcp103 through 192.168.206.27@tcp has gone from up to down [ 596.372635] LNetError: 33322:0:(lib-lnet.h:1279:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 596.373554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.206.[23-28/2]@tcp [ 602.126928] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 18:32:29 (1786746749) [ 602.211473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 602.248055] LNetError: 34022:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 602.275172] LNet: Removed LNI 192.168.206.22@tcp [ 602.299646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 602.384956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 602.426688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 602.461146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 602.534343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 602.598299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 602.631343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 602.687387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 602.787107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 602.828293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 602.901824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 602.983952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 603.020445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 603.064860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 603.150522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 603.189578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 603.248844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 603.314880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 603.359494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 603.441650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 603.585485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 603.610574] LNet: 34121:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 603.616327] LNet: 34121:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 603.641860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 603.727337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 603.807374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 603.835874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 603.887892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 604.036428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 604.096152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 604.162115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 604.307645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 604.361980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 604.428931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 604.487776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 604.529758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 604.600746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 609.932644] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 18:32:36 (1786746756) [ 611.534379] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg622-server [ 612.992586] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 18:32:40 (1786746760) [ 614.684643] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg622-server [ 616.146697] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 18:32:43 (1786746763) [ 616.850142] Key type .llcrypt unregistered [ 616.851580] Key type ._llcrypt unregistered [ 621.657877] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 621.787207] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 621.790225] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 621.793386] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 624.432699] Key type ._llcrypt registered [ 624.437541] Key type .llcrypt registered [ 624.634350] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 624.646641] alg: No test for adler32 (adler32-zlib) [ 625.586489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 625.647709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 625.673620] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 626.434694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 626.456505] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 628.457313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 628.493197] LNet: Removed LNI 10.1.2.3@tcp [ 628.618488] LNet: 36153:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 628.626458] LNetError: 36153:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 628.647314] LNet: Removed LNI 192.168.206.22@tcp [ 629.223604] Key type .llcrypt unregistered [ 629.230099] Key type ._llcrypt unregistered [ 637.978030] Key type ._llcrypt registered [ 637.980783] Key type .llcrypt registered [ 639.067796] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 18:33:06 (1786746786) [ 639.493480] Key type .llcrypt unregistered [ 639.505235] Key type ._llcrypt unregistered [ 644.622347] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 644.709972] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 644.715178] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 644.723040] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 647.272259] Key type ._llcrypt registered [ 647.274178] Key type .llcrypt registered [ 647.434276] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 647.445515] alg: No test for adler32 (adler32-zlib) [ 648.348541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 648.420780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 648.457627] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 649.133522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 649.167905] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 651.103279] LNet: 37652:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 651.109718] LNetError: 37652:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 651.117988] LNet: Removed LNI 10.1.2.3@tcp [ 651.130472] LNet: Removed LNI 192.168.206.22@tcp [ 651.674976] Key type .llcrypt unregistered [ 651.677290] Key type ._llcrypt unregistered [ 660.821948] Key type ._llcrypt registered [ 660.826633] Key type .llcrypt registered [ 661.965116] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 18:33:28 (1786746808) [ 662.398323] Key type .llcrypt unregistered [ 662.400518] Key type ._llcrypt unregistered [ 667.030820] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 667.150844] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 667.156173] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 667.162075] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 669.742335] Key type ._llcrypt registered [ 669.744548] Key type .llcrypt registered [ 669.930602] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 669.948820] alg: No test for adler32 (adler32-zlib) [ 670.878423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 670.967271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 670.990096] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 672.876643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 672.914766] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 674.501754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 674.521285] LNet: Removed LNI 10.1.2.3@tcp [ 674.531744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 674.559199] LNet: Removed LNI 10.1.2.31@tcp [ 674.573636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 674.590354] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 675.315062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 675.338435] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 676.200343] LNet: 39580:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 676.212370] LNetError: 39580:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 676.226112] LNet: Removed LNI 10.1.2.31@tcp [ 676.240023] LNet: Removed LNI 10.1.2.3@tcp [ 676.989683] Key type .llcrypt unregistered [ 676.994734] Key type ._llcrypt unregistered [ 687.013693] Key type ._llcrypt registered [ 687.015723] Key type .llcrypt registered [ 688.157949] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 18:33:55 (1786746835) [ 688.468291] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 688.483900] alg: No test for adler32 (adler32-zlib) [ 689.428904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 689.487782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 689.505914] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 690.202433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 690.232214] LNet: 40536:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 690.243838] LNetError: 40536:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 690.259100] LNet: Removed LNI 192.168.206.22@tcp [ 690.283631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 690.335451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.206.22@tcp [ 690.358778] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 690.959446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.206.22@tcp [ 690.989698] LNet: Removed LNI 192.168.206.22@tcp [ 691.070373] LNet: 40671:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 691.590728] Key type .llcrypt unregistered [ 691.592783] Key type ._llcrypt unregistered [ 701.203915] Key type ._llcrypt registered [ 701.205790] Key type .llcrypt registered [ 702.270457] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 18:34:09 (1786746849) [ 703.763578] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg622-server [ 705.100907] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 18:34:12 (1786746852) [ 705.564274] Key type .llcrypt unregistered [ 705.567439] Key type ._llcrypt unregistered [ 710.437362] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 710.556825] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 710.562895] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 710.571119] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 712.959036] Key type ._llcrypt registered [ 712.961948] Key type .llcrypt registered [ 713.101317] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 713.126861] alg: No test for adler32 (adler32-zlib) [ 714.019706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 714.077964] LNet: Added LNI 192.168.206.22@tcp1 [8/256/0/180] [ 714.663328] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 716.508968] LNet: 42510:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 716.519868] LNetError: 42510:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 716.527778] LNet: Removed LNI 192.168.206.22@tcp1 [ 716.550565] LNet: Removed LNI 10.1.2.3@tcp2 [ 717.212464] Key type .llcrypt unregistered [ 717.216571] Key type ._llcrypt unregistered [ 726.196960] Key type ._llcrypt registered [ 726.200905] Key type .llcrypt registered [ 727.252772] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 18:34:34 (1786746874) [ 727.607186] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 727.628166] alg: No test for adler32 (adler32-zlib) [ 728.540205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 728.605882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 728.630915] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 729.347904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 729.374068] LNet: 43482:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 729.393376] LNetError: 43482:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 729.408766] LNet: Removed LNI 192.168.206.22@tcp [ 729.429775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 729.487945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 729.513965] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 735.274502] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 18:34:42 (1786746882) [ 735.359573] LNet: 44065:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 735.369345] LNetError: 44065:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 735.395394] LNet: Removed LNI 192.168.206.22@tcp [ 735.984391] Key type .llcrypt unregistered [ 735.986853] Key type ._llcrypt unregistered [ 741.269213] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 741.414743] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 741.418214] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 741.421271] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 742.842332] Key type ._llcrypt registered [ 742.844125] Key type .llcrypt registered [ 743.030700] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 743.047280] alg: No test for adler32 (adler32-zlib) [ 743.939480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 743.981759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 743.998903] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 744.472473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 744.487195] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 746.485454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 746.503396] LNet: 44806:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 746.516637] LNetError: 44806:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 746.532063] LNet: Removed LNI 10.1.2.3@tcp [ 746.554710] LNet: Removed LNI 192.168.206.22@tcp [ 746.609812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 746.673682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 746.718363] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 746.728315] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 747.276695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 747.293299] LNet: 44833:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 747.306703] LNetError: 44833:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 747.312457] LNetError: 44833:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 747.318885] LNet: Removed LNI 10.1.2.3@tcp [ 747.328995] LNet: Removed LNI 192.168.206.22@tcp [ 747.354796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 747.428818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 747.466577] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 747.476597] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 747.858644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 747.874443] LNet: Added LNI 192.168.206.22@tcp2 [8/256/0/180] [ 748.686779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 748.716251] LNet: 44995:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 748.734461] LNetError: 44995:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 748.748360] LNetError: 44995:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 748.766332] LNet: Removed LNI 10.1.2.3@tcp [ 748.770523] LNet: Removed LNI 192.168.206.22@tcp [ 748.783972] LNet: Removed LNI 192.168.206.22@tcp2 [ 748.828385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 748.911749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 748.953464] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 748.956972] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 748.961250] LNet: Added LNI 192.168.206.22@tcp2 [8/256/0/180] [ 754.989547] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 18:35:01 (1786746901) [ 755.213477] LNet: 45593:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 755.218906] LNetError: 45593:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 755.226180] LNetError: 45593:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 755.232744] LNet: Removed LNI 10.1.2.3@tcp [ 755.235531] LNet: Removed LNI 192.168.206.22@tcp [ 755.246310] LNet: Removed LNI 192.168.206.22@tcp2 [ 755.981292] Key type .llcrypt unregistered [ 755.984386] Key type ._llcrypt unregistered [ 764.039855] Key type ._llcrypt registered [ 764.041850] Key type .llcrypt registered [ 764.258064] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 764.271893] alg: No test for adler32 (adler32-zlib) [ 765.273871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 765.343385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 765.366170] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 766.179200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 766.209500] LNet: 45991:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 766.215544] LNetError: 45991:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 766.234164] LNet: Removed LNI 192.168.206.22@tcp [ 766.253717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 766.326641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 766.362804] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 766.979335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 766.999602] LNet: 46021:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 767.008480] LNetError: 46021:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 767.024581] LNet: Removed LNI 192.168.206.22@tcp [ 767.048377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 767.100256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 767.121370] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 767.682456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 767.726370] LNet: Removed LNI 192.168.206.22@tcp [ 767.796226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 767.918483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 767.974425] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 768.304985] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 768.478722] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 768.490211] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 768.498147] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 770.910770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 770.933846] LNet: 46131:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 770.939564] LNet: 46131:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 770.951694] LNetError: 46131:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 770.957740] LNetError: 46131:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 770.971641] LNet: Removed LNI 192.168.206.22@tcp [ 770.988934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 771.058611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 771.084332] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 777.505878] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 18:35:24 (1786746924) [ 777.680439] LNet: 46722:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 777.691031] LNetError: 46722:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 777.702819] LNet: Removed LNI 192.168.206.22@tcp [ 778.374313] Key type .llcrypt unregistered [ 778.380395] Key type ._llcrypt unregistered [ 784.814948] Key type ._llcrypt registered [ 784.819389] Key type .llcrypt registered [ 785.037873] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 785.063752] alg: No test for adler32 (adler32-zlib) [ 785.971619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 786.043811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 786.081363] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 786.874526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 786.891933] LNet: 47130:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 786.900153] LNetError: 47130:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 786.917837] LNet: Removed LNI 192.168.206.22@tcp [ 786.949817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 786.996911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 787.015957] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 794.369569] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 18:35:40 (1786746940) [ 794.809786] LNet: 47722:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 794.829397] LNetError: 47722:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 794.854745] LNet: Removed LNI 192.168.206.22@tcp [ 795.819564] Key type .llcrypt unregistered [ 795.821862] Key type ._llcrypt unregistered [ 803.356500] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 803.444149] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 803.447171] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 803.450347] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 806.008276] Key type ._llcrypt registered [ 806.009914] Key type .llcrypt registered [ 806.261234] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 806.299266] alg: No test for adler32 (adler32-zlib) [ 807.232435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 807.317594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 807.352501] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 808.326253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 808.347928] LNet: 48178:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 808.355270] LNetError: 48178:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 808.378864] LNet: Removed LNI 192.168.206.22@tcp [ 808.406456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 808.490631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 808.518054] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 814.825250] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 18:36:01 (1786746961) [ 815.005049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 815.024893] LNet: 48776:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 815.031871] LNetError: 48776:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 815.056603] LNet: Removed LNI 192.168.206.22@tcp [ 815.086100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 815.278959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 820.816462] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 18:36:07 (1786746967) [ 820.884343] LNet: 49350:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 821.450890] Key type .llcrypt unregistered [ 821.454840] Key type ._llcrypt unregistered [ 826.953673] Key type ._llcrypt registered [ 826.958921] Key type .llcrypt registered [ 827.295791] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 827.341685] alg: No test for adler32 (adler32-zlib) [ 828.249248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 828.301389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 828.324588] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 829.047901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 829.061793] LNet: 49761:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 829.066828] LNetError: 49761:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 829.079846] LNet: Removed LNI 192.168.206.22@tcp [ 829.102776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 829.155360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 829.174066] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 834.655784] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 18:36:21 (1786746981) [ 834.848147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 834.873795] LNet: 50354:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 834.880310] LNetError: 50354:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 834.896469] LNet: Removed LNI 192.168.206.22@tcp [ 834.926734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 835.015782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 840.557985] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 18:36:27 (1786746987) [ 841.716348] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 842.994334] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 18:36:30 (1786746990) [ 844.183668] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 845.537616] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 18:36:32 (1786746992) [ 846.815935] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 848.210740] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 18:36:35 (1786746995) [ 849.368649] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 850.572582] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 18:36:37 (1786746997) [ 851.917820] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 853.235691] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 18:36:40 (1786747000) [ 853.364386] LNet: 52404:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 853.916564] Key type .llcrypt unregistered [ 853.920726] Key type ._llcrypt unregistered [ 859.516350] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 859.586456] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 859.589913] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 859.595504] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 862.164536] Key type ._llcrypt registered [ 862.171042] Key type .llcrypt registered [ 862.373349] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 862.383615] alg: No test for adler32 (adler32-zlib) [ 863.356651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 863.445610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 863.471948] LNet: Added LNI 192.168.206.22@tcp [8/512/0/180] [ 863.479625] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 863.483351] LNet: 52729:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 863.582575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 863.604644] LNet: 52748:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 863.610911] LNetError: 52748:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 863.623153] LNet: Removed LNI 10.1.2.3@tcp [ 863.626661] LNetError: 52748:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 863.642889] LNet: Removed LNI 192.168.206.22@tcp [ 863.680814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 863.729686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 863.755473] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 863.810530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 863.827088] LNet: 52772:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 863.832846] LNetError: 52772:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 863.844850] LNet: Removed LNI 192.168.206.22@tcp [ 863.865858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 863.925993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 863.947563] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 863.952815] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 863.956443] LNet: 52780:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 864.041543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 864.061591] LNet: 52798:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 864.072558] LNetError: 52798:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 864.078418] LNet: Removed LNI 10.1.2.3@tcp [ 864.081416] LNetError: 52798:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 864.096359] LNet: Removed LNI 192.168.206.22@tcp [ 864.115550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 864.205694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 864.261572] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 864.265212] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 864.268611] LNet: 52806:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 864.480675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 864.498521] LNet: 52832:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 864.505878] LNetError: 52832:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 864.511287] LNet: Removed LNI 10.1.2.3@tcp [ 864.514886] LNetError: 52832:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 864.556330] LNet: Removed LNI 192.168.206.22@tcp [ 864.600100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 864.681456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 864.704351] LNet: Added LNI 192.168.206.22@tcp [8/256/0/182] [ 864.723857] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 864.813695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 864.832762] LNet: 52860:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 864.838348] LNetError: 52860:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 864.844120] LNet: Removed LNI 10.1.2.3@tcp [ 864.846853] LNetError: 52860:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 864.867183] LNet: Removed LNI 192.168.206.22@tcp [ 864.905591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 865.000504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 865.029864] LNet: Added LNI 192.168.206.22@tcp [8/256/0/182] [ 865.034980] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 865.041172] LNet: 52868:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 865.140277] LNet: Removed LNI 10.1.2.3@tcp [ 865.166580] LNet: Removed LNI 192.168.206.22@tcp [ 865.994349] Key type .llcrypt unregistered [ 866.000695] Key type ._llcrypt unregistered [ 878.223726] Key type ._llcrypt registered [ 878.226365] Key type .llcrypt registered [ 879.828525] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 18:37:06 (1786747026) [ 881.312603] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 882.911778] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 18:37:09 (1786747029) [ 883.258109] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 883.272557] alg: No test for adler32 (adler32-zlib) [ 884.173618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 884.209542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 884.224419] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 885.041640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.206.22 [ 885.072914] LNet: 54137:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 885.089290] LNet: Removed LNI 192.168.206.22@tcp [ 891.997808] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 18:37:18 (1786747038) [ 892.123691] LNet: 54698:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 892.133946] LNetError: 54698:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 892.180763] LNet: Removed LNI 192.168.206.22@tcp [ 892.871700] Key type .llcrypt unregistered [ 892.877145] Key type ._llcrypt unregistered [ 899.104254] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 899.326488] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 902.367604] Key type ._llcrypt registered [ 902.370905] Key type .llcrypt registered [ 902.640298] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 902.677899] alg: No test for adler32 (adler32-zlib) [ 903.971561] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 910.867574] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 18:37:37 (1786747057) [ 911.033286] LNet: 55617:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 911.056050] LNetError: 55617:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 911.084198] LNet: Removed LNI 10.1.2.3@tcp [ 912.131873] Key type .llcrypt unregistered [ 912.140193] Key type ._llcrypt unregistered [ 919.471635] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 919.647450] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 919.655071] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 919.663434] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 922.356987] Key type ._llcrypt registered [ 922.362989] Key type .llcrypt registered [ 922.567541] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 922.578975] alg: No test for adler32 (adler32-zlib) [ 923.663366] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 930.081596] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 18:37:56 (1786747076) [ 931.617936] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 933.295615] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 18:38:00 (1786747080) [ 933.692680] LNet: 56815:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 933.705391] LNetError: 56815:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 933.721602] LNet: Removed LNI 10.1.2.3@tcp [ 934.316861] Key type .llcrypt unregistered [ 934.320458] Key type ._llcrypt unregistered [ 939.779416] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 940.022666] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 940.027420] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 940.036757] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 942.841690] Key type ._llcrypt registered [ 942.846218] Key type .llcrypt registered [ 943.140082] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 943.209798] alg: No test for adler32 (adler32-zlib) [ 944.198424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 944.287666] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 953.574262] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 18:38:20 (1786747100) [ 953.885847] LNet: 58048:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 953.893973] LNetError: 58048:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 953.923265] LNet: Removed LNI 10.1.2.3@tcp [ 954.490485] Key type .llcrypt unregistered [ 954.494056] Key type ._llcrypt unregistered [ 960.375478] Key type ._llcrypt registered [ 960.377506] Key type .llcrypt registered [ 960.553249] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 960.571770] alg: No test for adler32 (adler32-zlib) [ 961.472504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 961.500537] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 968.280953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.122@tcp [ 968.844878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.22@tcp -d 192.168.206.122@tcp -m GET -e local_interrupt -r 1 [ 968.885477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.22@tcp -d 192.168.206.22@tcp -m GET -e local_interrupt -r 1 [ 968.918679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.122@tcp [ 968.933136] LNet: There was an unexpected network error while writing to 192.168.206.122: rc = -22 [ 969.156890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 969.190922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 969.230954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 969.258225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 969.287353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 969.315517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 969.358391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 969.389328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 969.406839] LNet: 58489:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 969.445234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 969.487252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 969.519880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 969.557926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 969.596691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 969.632932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 969.657448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 970.149277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.22@tcp -d 192.168.206.122@tcp -m GET -e local_dropped -r 1 [ 970.192907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.22@tcp -d 192.168.206.22@tcp -m GET -e local_dropped -r 1 [ 970.231741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.122@tcp [ 970.270666] LNet: There was an unexpected network error while writing to 192.168.206.122: rc = -22 [ 970.471870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 970.518505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 970.559922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 970.598744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 970.644060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 970.704212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 970.721635] LNet: 1 local NIs in recovery (showing 1): 192.168.206.22@tcp [ 970.750949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 970.813995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 970.856816] LNet: 58598:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 970.877740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 970.919553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 970.950966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 970.981174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 971.012091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 971.037778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 971.069690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 971.562891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.22@tcp -d 192.168.206.122@tcp -m GET -e local_aborted -r 1 [ 971.599040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.22@tcp -d 192.168.206.22@tcp -m GET -e local_aborted -r 1 [ 971.631928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.122@tcp [ 971.651843] LNet: There was an unexpected network error while writing to 192.168.206.122: rc = -22 [ 971.883235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 971.913577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 971.952822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 971.983898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 972.009832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 972.049451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 972.089537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 972.118440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 972.137413] LNet: 58708:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 972.152954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 972.180541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 972.212387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 972.238270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 972.270658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 972.313483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 972.350255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 972.810513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.22@tcp -d 192.168.206.122@tcp -m GET -e local_no_route -r 1 [ 972.841173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.22@tcp -d 192.168.206.22@tcp -m GET -e local_no_route -r 1 [ 972.868702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.122@tcp [ 973.176539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 973.200861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 973.238630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 973.274362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 973.314789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 973.345247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 973.372285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 973.403651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 973.434301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 973.467895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 973.509391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 973.537264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 973.567964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 973.600159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 973.632411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 974.075167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.22@tcp -d 192.168.206.122@tcp -m GET -e local_timeout -r 1 [ 974.099370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.22@tcp -d 192.168.206.22@tcp -m GET -e local_timeout -r 1 [ 974.130362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.122@tcp [ 974.147184] LNet: There was an unexpected network error while writing to 192.168.206.122: rc = -22 [ 974.152291] LNet: Skipped 2 previous similar messages [ 974.331538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 974.366801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 974.397825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 974.428921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 974.460682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 974.504788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 974.538649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 974.569169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 974.586820] LNet: 58926:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 974.594391] LNet: 58926:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 974.608887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 974.637795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 974.671173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 974.695110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 974.722250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 974.744707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 974.768208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 975.223317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.22@tcp -d 192.168.206.122@tcp -m GET -e local_error -r 1 [ 975.252246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.22@tcp -d 192.168.206.22@tcp -m GET -e local_error -r 1 [ 975.277702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.122@tcp [ 975.447838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 975.471272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 975.508513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 975.531324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 975.560622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 975.585540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 975.634200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 975.666207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 975.691417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 975.720347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 975.750686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 975.783547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 975.818328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 975.859512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 975.897221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 976.232023] LNet: 59080:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 976.240617] LNetError: 59080:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 976.252522] LNet: Removed LNI 192.168.206.22@tcp [ 976.633727] Key type .llcrypt unregistered [ 976.637028] Key type ._llcrypt unregistered [ 991.498323] Key type ._llcrypt registered [ 991.500335] Key type .llcrypt registered [ 992.673299] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 18:38:59 (1786747139) [ 993.423278] Key type .llcrypt unregistered [ 993.427196] Key type ._llcrypt unregistered [ 998.315264] Key type ._llcrypt registered [ 998.317393] Key type .llcrypt registered [ 998.458351] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 998.475101] alg: No test for adler32 (adler32-zlib) [ 999.417281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 999.455048] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 1005.828489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.122@tcp [ 1014.785353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1014.840115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1014.859866] LNet: Added LNI 192.168.206.22@tcp1 [8/256/0/180] [ 1015.937432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.22@tcp -d 192.168.206.122@tcp -m GET -e local_interrupt -r 1 [ 1015.969448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.22@tcp -d 192.168.206.122@tcp1 -m GET -e local_interrupt -r 1 [ 1016.017656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.22@tcp -d 192.168.206.22@tcp -m GET -e local_interrupt -r 1 [ 1016.057954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.22@tcp -d 192.168.206.22@tcp1 -m GET -e local_interrupt -r 1 [ 1016.088507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.22@tcp1 -d 192.168.206.122@tcp -m GET -e local_interrupt -r 1 [ 1016.127845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.22@tcp1 -d 192.168.206.122@tcp1 -m GET -e local_interrupt -r 1 [ 1016.156293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.22@tcp1 -d 192.168.206.22@tcp -m GET -e local_interrupt -r 1 [ 1016.188450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.22@tcp1 -d 192.168.206.22@tcp1 -m GET -e local_interrupt -r 1 [ 1016.243825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.122@tcp [ 1016.270266] LNet: There was an unexpected network error while writing to 192.168.206.122: rc = -22 [ 1016.274954] LNet: 1 local NIs in recovery (showing 1): 192.168.206.22@tcp [ 1016.469997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1016.501836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1016.554101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1016.593179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1016.623122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1016.646980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1016.678771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1016.712850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1016.732868] LNet: 60550:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1016.748182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1016.775823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1016.801599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1016.829518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1016.859764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1016.892861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1016.933792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1017.392566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.22@tcp -d 192.168.206.122@tcp -m GET -e local_dropped -r 1 [ 1017.414979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.22@tcp -d 192.168.206.122@tcp1 -m GET -e local_dropped -r 1 [ 1017.445133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.22@tcp -d 192.168.206.22@tcp -m GET -e local_dropped -r 1 [ 1017.475929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.22@tcp -d 192.168.206.22@tcp1 -m GET -e local_dropped -r 1 [ 1017.502936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.22@tcp1 -d 192.168.206.122@tcp -m GET -e local_dropped -r 1 [ 1017.534122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.22@tcp1 -d 192.168.206.122@tcp1 -m GET -e local_dropped -r 1 [ 1017.559113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.22@tcp1 -d 192.168.206.22@tcp -m GET -e local_dropped -r 1 [ 1017.584733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.22@tcp1 -d 192.168.206.22@tcp1 -m GET -e local_dropped -r 1 [ 1017.611364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.122@tcp [ 1017.626210] LNet: There was an unexpected network error while writing to 192.168.206.122: rc = -22 [ 1017.630859] LNet: Skipped 2 previous similar messages [ 1017.777466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1017.808381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1017.846323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1017.877173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1017.911185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1017.943168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1017.995714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1018.030452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1018.054502] LNet: 60673:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1018.083991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1018.128250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1018.175384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1018.216916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1018.257506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1018.292731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1018.315540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1018.815498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.22@tcp -d 192.168.206.122@tcp -m GET -e local_aborted -r 1 [ 1018.869598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.22@tcp -d 192.168.206.122@tcp1 -m GET -e local_aborted -r 1 [ 1018.927407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.22@tcp -d 192.168.206.22@tcp -m GET -e local_aborted -r 1 [ 1018.983917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.22@tcp -d 192.168.206.22@tcp1 -m GET -e local_aborted -r 1 [ 1019.038947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.22@tcp1 -d 192.168.206.122@tcp -m GET -e local_aborted -r 1 [ 1019.089983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.22@tcp1 -d 192.168.206.122@tcp1 -m GET -e local_aborted -r 1 [ 1019.128464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.22@tcp1 -d 192.168.206.22@tcp -m GET -e local_aborted -r 1 [ 1019.168866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.22@tcp1 -d 192.168.206.22@tcp1 -m GET -e local_aborted -r 1 [ 1019.208559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.122@tcp [ 1019.229899] LNet: There was an unexpected network error while writing to 192.168.206.122: rc = -22 [ 1019.236947] LNet: Skipped 2 previous similar messages [ 1019.438988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1019.465921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1019.488754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1019.514746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1019.541956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1019.574040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1019.610328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1019.638836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1019.654537] LNet: 60794:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1019.668419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1019.697807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1019.729236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1019.756956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1019.783724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1019.823777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1019.860155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1020.388799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.22@tcp -d 192.168.206.122@tcp -m GET -e local_no_route -r 1 [ 1020.446229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.22@tcp -d 192.168.206.122@tcp1 -m GET -e local_no_route -r 1 [ 1020.478293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.22@tcp -d 192.168.206.22@tcp -m GET -e local_no_route -r 1 [ 1020.514252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.22@tcp -d 192.168.206.22@tcp1 -m GET -e local_no_route -r 1 [ 1020.549923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.22@tcp1 -d 192.168.206.122@tcp -m GET -e local_no_route -r 1 [ 1020.594816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.22@tcp1 -d 192.168.206.122@tcp1 -m GET -e local_no_route -r 1 [ 1020.647483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.22@tcp1 -d 192.168.206.22@tcp -m GET -e local_no_route -r 1 [ 1020.728929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.22@tcp1 -d 192.168.206.22@tcp1 -m GET -e local_no_route -r 1 [ 1020.780456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.122@tcp [ 1021.005400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1021.033349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1021.065492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1021.094063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1021.137782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1021.185626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1021.218336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1021.246903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1021.292787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1021.325676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1021.354570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1021.394922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1021.425348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1021.459877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1021.500090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1021.930277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.22@tcp -d 192.168.206.122@tcp -m GET -e local_timeout -r 1 [ 1021.966828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.22@tcp -d 192.168.206.122@tcp1 -m GET -e local_timeout -r 1 [ 1022.004427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.22@tcp -d 192.168.206.22@tcp -m GET -e local_timeout -r 1 [ 1022.031982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.22@tcp -d 192.168.206.22@tcp1 -m GET -e local_timeout -r 1 [ 1022.059909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.22@tcp1 -d 192.168.206.122@tcp -m GET -e local_timeout -r 1 [ 1022.086986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.22@tcp1 -d 192.168.206.122@tcp1 -m GET -e local_timeout -r 1 [ 1022.113327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.22@tcp1 -d 192.168.206.22@tcp -m GET -e local_timeout -r 1 [ 1022.157870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.22@tcp1 -d 192.168.206.22@tcp1 -m GET -e local_timeout -r 1 [ 1022.204268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.122@tcp [ 1022.231939] LNet: There was an unexpected network error while writing to 192.168.206.122: rc = -22 [ 1022.239797] LNet: Skipped 5 previous similar messages [ 1022.459074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1022.491540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1022.517274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1022.554562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1022.590994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1022.615965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1022.644231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1022.678742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1022.691619] LNet: 61036:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1022.696884] LNet: 61036:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1022.718549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1022.754990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1022.783674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1022.807214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1022.832733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1022.858587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1022.890950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1023.325854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.22@tcp -d 192.168.206.122@tcp -m GET -e local_error -r 1 [ 1023.354960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.22@tcp -d 192.168.206.122@tcp1 -m GET -e local_error -r 1 [ 1023.384796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.22@tcp -d 192.168.206.22@tcp -m GET -e local_error -r 1 [ 1023.439415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.22@tcp -d 192.168.206.22@tcp1 -m GET -e local_error -r 1 [ 1023.477062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.22@tcp1 -d 192.168.206.122@tcp -m GET -e local_error -r 1 [ 1023.518156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.22@tcp1 -d 192.168.206.122@tcp1 -m GET -e local_error -r 1 [ 1023.545122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.22@tcp1 -d 192.168.206.22@tcp -m GET -e local_error -r 1 [ 1023.570705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.22@tcp1 -d 192.168.206.22@tcp1 -m GET -e local_error -r 1 [ 1023.597761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.122@tcp [ 1023.827752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1023.853431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1023.877556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1023.903534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1023.946028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1023.984354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1024.037949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1024.084563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1024.119879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1024.145238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1024.190560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1024.224063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1024.254717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1024.283721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1024.313596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1028.806068] LNet: 61237:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1028.825569] LNetError: 61237:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1028.839286] LNet: Removed LNI 192.168.206.22@tcp [ 1029.864613] LNet: Removed LNI 192.168.206.22@tcp1 [ 1030.303875] Key type .llcrypt unregistered [ 1030.310936] Key type ._llcrypt unregistered [ 1044.950643] Key type ._llcrypt registered [ 1044.956099] Key type .llcrypt registered [ 1046.195144] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 18:39:53 (1786747193) [ 1046.898793] Key type .llcrypt unregistered [ 1046.904080] Key type ._llcrypt unregistered [ 1052.189743] Key type ._llcrypt registered [ 1052.192364] Key type .llcrypt registered [ 1052.379858] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1052.404164] alg: No test for adler32 (adler32-zlib) [ 1053.275628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1053.298798] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 1059.313360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.122@tcp [ 1059.807694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.22@tcp -d 192.168.206.122@tcp -m GET -e remote_dropped -r 1 [ 1059.842997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.22@tcp -d 192.168.206.22@tcp -m GET -e remote_dropped -r 1 [ 1059.889125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.122@tcp [ 1059.907848] LNet: There was an unexpected network error while writing to 192.168.206.122: rc = -22 [ 1060.131647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1060.158543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1060.182519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1060.213601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1060.247343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1060.289187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1060.311158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1060.344736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1060.367503] LNet: 62490:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1060.391773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1060.433159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1060.464641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1060.495898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1060.532153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1060.567856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1060.599548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1061.054622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.22@tcp -d 192.168.206.122@tcp -m GET -e remote_error -r 1 [ 1061.081159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.22@tcp -d 192.168.206.22@tcp -m GET -e remote_error -r 1 [ 1061.112507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.122@tcp [ 1061.128404] LNet: There was an unexpected network error while writing to 192.168.206.122: rc = -22 [ 1061.317864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1061.338711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1061.379990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1061.412810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1061.438290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1061.477153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1061.507897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1061.549119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1061.578059] LNet: 62599:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1061.595868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1061.631161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1061.659801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1061.689355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1061.720095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1061.750920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1061.775900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1062.136680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.22@tcp -d 192.168.206.122@tcp -m GET -e remote_timeout -r 1 [ 1062.176775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.22@tcp -d 192.168.206.22@tcp -m GET -e remote_timeout -r 1 [ 1062.205626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.122@tcp [ 1062.221882] LNet: There was an unexpected network error while writing to 192.168.206.122: rc = -22 [ 1062.385307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1062.416498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1062.444753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1062.473535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1062.507842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1062.552027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1062.578915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1062.601362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1062.614832] LNet: 62708:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1062.628876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1062.668595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1062.703082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1062.736340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1062.779564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1062.817430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1062.888733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1063.215062] LNet: 62753:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1063.220663] LNetError: 62753:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1064.298795] LNet: Removed LNI 192.168.206.22@tcp [ 1064.700721] Key type .llcrypt unregistered [ 1064.703115] Key type ._llcrypt unregistered [ 1077.095130] Key type ._llcrypt registered [ 1077.096878] Key type .llcrypt registered [ 1078.191670] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 18:40:25 (1786747225) [ 1078.837556] Key type .llcrypt unregistered [ 1078.841885] Key type ._llcrypt unregistered [ 1084.185207] Key type ._llcrypt registered [ 1084.188371] Key type .llcrypt registered [ 1084.499919] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1084.518824] alg: No test for adler32 (adler32-zlib) [ 1085.383762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1085.404566] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 1091.944677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.122@tcp [ 1100.029708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1100.092120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1100.112511] LNet: Added LNI 192.168.206.22@tcp1 [8/256/0/180] [ 1101.599424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.22@tcp -d 192.168.206.122@tcp -m GET -e remote_dropped -r 1 [ 1101.628353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.22@tcp -d 192.168.206.122@tcp1 -m GET -e remote_dropped -r 1 [ 1101.666801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.22@tcp -d 192.168.206.22@tcp -m GET -e remote_dropped -r 1 [ 1101.727985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.22@tcp -d 192.168.206.22@tcp1 -m GET -e remote_dropped -r 1 [ 1101.761883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.22@tcp1 -d 192.168.206.122@tcp -m GET -e remote_dropped -r 1 [ 1101.814447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.22@tcp1 -d 192.168.206.122@tcp1 -m GET -e remote_dropped -r 1 [ 1101.840721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.22@tcp1 -d 192.168.206.22@tcp -m GET -e remote_dropped -r 1 [ 1101.871630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.22@tcp1 -d 192.168.206.22@tcp1 -m GET -e remote_dropped -r 1 [ 1101.901239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.122@tcp [ 1101.916156] LNet: There was an unexpected network error while writing to 192.168.206.122: rc = -22 [ 1101.919328] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.122@tcp [ 1102.113627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1102.147495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1102.191889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1102.234229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1102.267991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1102.293928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1102.322942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1102.351556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1102.364792] LNet: 64222:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1102.377163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1102.403482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1102.424632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1102.447491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1102.471754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1102.498345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1102.537602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1102.569610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1103.229336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.22@tcp -d 192.168.206.122@tcp -m GET -e remote_error -r 1 [ 1103.285465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.22@tcp -d 192.168.206.122@tcp1 -m GET -e remote_error -r 1 [ 1103.349831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.22@tcp -d 192.168.206.22@tcp -m GET -e remote_error -r 1 [ 1103.391058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.22@tcp -d 192.168.206.22@tcp1 -m GET -e remote_error -r 1 [ 1103.446795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.22@tcp1 -d 192.168.206.122@tcp -m GET -e remote_error -r 1 [ 1103.511665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.22@tcp1 -d 192.168.206.122@tcp1 -m GET -e remote_error -r 1 [ 1103.567080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.22@tcp1 -d 192.168.206.22@tcp -m GET -e remote_error -r 1 [ 1103.618395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.22@tcp1 -d 192.168.206.22@tcp1 -m GET -e remote_error -r 1 [ 1103.667479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.122@tcp [ 1103.694146] LNet: There was an unexpected network error while writing to 192.168.206.122: rc = -22 [ 1103.706506] LNet: Skipped 2 previous similar messages [ 1103.963456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1103.990271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1104.015500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1104.045657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1104.069343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1104.092457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1104.121462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1104.147440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1104.167690] LNet: 64345:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1104.182706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1104.209754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1104.249049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1104.274120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1104.304189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1104.331888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1104.359500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1104.385312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1104.842666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.22@tcp -d 192.168.206.122@tcp -m GET -e remote_timeout -r 1 [ 1104.878672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.22@tcp -d 192.168.206.122@tcp1 -m GET -e remote_timeout -r 1 [ 1104.922384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.22@tcp -d 192.168.206.22@tcp -m GET -e remote_timeout -r 1 [ 1104.947785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.22@tcp -d 192.168.206.22@tcp1 -m GET -e remote_timeout -r 1 [ 1104.976779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.22@tcp1 -d 192.168.206.122@tcp -m GET -e remote_timeout -r 1 [ 1105.003420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.22@tcp1 -d 192.168.206.122@tcp1 -m GET -e remote_timeout -r 1 [ 1105.029736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.22@tcp1 -d 192.168.206.22@tcp -m GET -e remote_timeout -r 1 [ 1105.066685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.22@tcp1 -d 192.168.206.22@tcp1 -m GET -e remote_timeout -r 1 [ 1105.110620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.122@tcp [ 1105.132908] LNet: There was an unexpected network error while writing to 192.168.206.122: rc = -22 [ 1105.290520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1105.318588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1105.352931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1105.386315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1105.412612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1105.436799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1105.466676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1105.489285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1105.506878] LNet: 64469:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1105.527613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1105.548259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1105.574255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1105.593940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1105.623703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1105.644592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1105.695351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1105.731940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1110.313166] LNet: 64550:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1110.324877] LNetError: 64550:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1110.335709] LNet: Removed LNI 192.168.206.22@tcp [ 1110.346470] LNet: Removed LNI 192.168.206.22@tcp1 [ 1110.805831] Key type .llcrypt unregistered [ 1110.808481] Key type ._llcrypt unregistered [ 1126.625161] Key type ._llcrypt registered [ 1126.627852] Key type .llcrypt registered [ 1127.772382] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 18:41:14 (1786747274) [ 1128.967959] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1130.530375] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 18:41:17 (1786747277) [ 1131.207588] Key type .llcrypt unregistered [ 1131.210749] Key type ._llcrypt unregistered [ 1135.660320] Key type ._llcrypt registered [ 1135.662546] Key type .llcrypt registered [ 1135.805469] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1135.815955] alg: No test for adler32 (adler32-zlib) [ 1136.673639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1136.698980] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 1142.737370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.122@tcp [ 1143.274409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.22@tcp -d 192.168.206.122@tcp -m GET -e network_timeout -r 1 [ 1143.303756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.22@tcp -d 192.168.206.22@tcp -m GET -e network_timeout -r 1 [ 1143.332499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.122@tcp [ 1143.347684] LNet: There was an unexpected network error while writing to 192.168.206.122: rc = -22 [ 1143.540974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1143.563790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1143.590848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1143.614973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1143.659205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1143.684198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1143.705093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1143.728457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1143.745467] LNet: 66098:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1143.763255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1143.789260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1143.813901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1143.844685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1143.872026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1143.899366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1143.904208] LNet: 1 local NIs in recovery (showing 1): 192.168.206.22@tcp [ 1144.119192] LNet: 66141:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1144.132265] LNetError: 66141:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1144.147943] LNet: Removed LNI 192.168.206.22@tcp [ 1144.428408] Key type .llcrypt unregistered [ 1144.431722] Key type ._llcrypt unregistered [ 1160.214720] Key type ._llcrypt registered [ 1160.216778] Key type .llcrypt registered [ 1160.403297] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1160.420529] alg: No test for adler32 (adler32-zlib) [ 1161.347903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1161.376182] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 1167.918150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.122@tcp [ 1175.392226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1175.426410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1175.454280] LNet: Added LNI 192.168.206.22@tcp1 [8/256/0/180] [ 1176.755975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.22@tcp -d 192.168.206.122@tcp -m GET -e network_timeout -r 1 [ 1176.801687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.22@tcp -d 192.168.206.122@tcp1 -m GET -e network_timeout -r 1 [ 1176.850861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.22@tcp -d 192.168.206.22@tcp -m GET -e network_timeout -r 1 [ 1176.895254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.22@tcp -d 192.168.206.22@tcp1 -m GET -e network_timeout -r 1 [ 1176.927514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.22@tcp1 -d 192.168.206.122@tcp -m GET -e network_timeout -r 1 [ 1176.958785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.22@tcp1 -d 192.168.206.122@tcp1 -m GET -e network_timeout -r 1 [ 1176.999500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.22@tcp1 -d 192.168.206.22@tcp -m GET -e network_timeout -r 1 [ 1177.031784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.22@tcp1 -d 192.168.206.22@tcp1 -m GET -e network_timeout -r 1 [ 1177.065891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.122@tcp [ 1177.085505] LNet: There was an unexpected network error while writing to 192.168.206.122: rc = -22 [ 1177.274611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1177.308737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1177.343919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1177.387423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1177.424578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1177.466976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1177.506550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1177.535820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1177.562966] LNet: 67024:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1177.590722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1177.624267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1177.653432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1177.680349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1177.709186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1177.752299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1177.761095] LNet: 1 local NIs in recovery (showing 1): 192.168.206.22@tcp [ 1181.992282] LNet: There was an unexpected network error while writing to 192.168.206.122: rc = -22 [ 1182.250358] LNet: 67101:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1182.260047] LNetError: 67101:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1182.270806] LNet: Removed LNI 192.168.206.22@tcp [ 1182.287640] LNet: Removed LNI 192.168.206.22@tcp1 [ 1182.580708] Key type .llcrypt unregistered [ 1182.585278] Key type ._llcrypt unregistered [ 1195.451337] Key type ._llcrypt registered [ 1195.453861] Key type .llcrypt registered [ 1196.619302] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 18:42:23 (1786747343) [ 1196.958576] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1196.972463] alg: No test for adler32 (adler32-zlib) [ 1197.860737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1197.924783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1197.945835] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 1198.546878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1198.563321] LNet: Added LNI 192.168.206.22@tcp1 [8/256/0/180] [ 1199.191404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.22@tcp [ 1199.268754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1199.320961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1199.360925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1199.429590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1199.465097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1199.504958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.22@tcp [ 1199.532236] LNetError: 68200:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.206.22@tcp to recovery [ 1199.968284] LNet: 1 local NIs in recovery (showing 1): 192.168.206.22@tcp [ 1199.968907] LNet: There was an unexpected network error while writing to 192.168.206.22: rc = -22 [ 1202.016370] LNet: There was an unexpected network error while writing to 192.168.206.22: rc = -22 [ 1202.961653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1206.112638] LNet: There was an unexpected network error while writing to 192.168.206.22: rc = -22 [ 1206.523629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1206.557669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1206.625623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1206.653189] LNet: 68307:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1206.659055] LNetError: 68307:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1206.670841] LNet: Removed LNI 192.168.206.22@tcp [ 1206.692921] LNet: Removed LNI 192.168.206.22@tcp1 [ 1206.714926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1206.769848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1206.805452] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 1207.389369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1207.412493] LNet: Added LNI 192.168.206.22@tcp1 [8/256/0/180] [ 1208.083908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.22@tcp [ 1208.125348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1208.176466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1208.224919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1208.273439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1208.314173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1208.361456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1208.398292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.22@tcp [ 1208.423258] LNetError: 68579:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.206.22@tcp to recovery [ 1208.805319] LNet: There was an unexpected network error while writing to 192.168.206.22: rc = -22 [ 1211.998840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1214.944689] LNet: There was an unexpected network error while writing to 192.168.206.22: rc = -22 [ 1214.954691] LNet: Skipped 1 previous similar message [ 1218.846280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1218.876549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1218.902784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1225.088516] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 18:42:51 (1786747371) [ 1225.174597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1225.195281] LNet: 69283:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1225.200524] LNetError: 69283:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1225.207827] LNetError: 69283:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1225.212538] LNet: Removed LNI 192.168.206.22@tcp [ 1225.220719] LNet: Removed LNI 192.168.206.22@tcp1 [ 1225.260562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1225.307551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1225.328895] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 1226.084833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1226.112296] LNet: Added LNI 192.168.206.22@tcp1 [8/256/0/180] [ 1226.976895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.22@tcp [ 1227.061378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1227.097964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1227.145854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1227.201381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1227.256086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1227.304818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.22@tcp --health 0 [ 1227.360108] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.22@tcp [ 1227.407863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1228.386897] LNet: There was an unexpected network error while writing to 192.168.206.22: rc = -22 [ 1228.401719] LNet: Skipped 1 previous similar message [ 1242.827679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1242.971513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1243.012898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1243.043177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.22@tcp --health 500 [ 1243.128479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1250.940602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1251.042746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1251.065966] LNet: 69786:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1251.073701] LNetError: 69786:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1251.078522] LNetError: 69786:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1251.084983] LNet: Removed LNI 192.168.206.22@tcp [ 1251.095808] LNet: Removed LNI 192.168.206.22@tcp1 [ 1251.134965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1251.218125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1251.254590] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 1252.027208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1252.065595] LNet: Added LNI 192.168.206.22@tcp1 [8/256/0/180] [ 1252.971222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.22@tcp [ 1253.028332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1253.114703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1253.189763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1253.236322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1253.271887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1253.303853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1253.345132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.22@tcp --health 0 [ 1254.240762] LNet: There was an unexpected network error while writing to 192.168.206.22: rc = -22 [ 1254.250126] LNet: Skipped 3 previous similar messages [ 1254.720581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1264.898162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1264.927358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1264.971367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1270.816928] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 18:43:37 (1786747417) [ 1271.250845] LNet: 70831:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1271.255964] LNetError: 70831:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1271.260092] LNetError: 70831:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1271.265750] LNet: Removed LNI 192.168.206.22@tcp [ 1271.278213] LNet: Removed LNI 192.168.206.22@tcp1 [ 1271.874400] Key type .llcrypt unregistered [ 1271.879738] Key type ._llcrypt unregistered [ 1277.455188] Key type ._llcrypt registered [ 1277.458230] Key type .llcrypt registered [ 1277.642629] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1277.656234] alg: No test for adler32 (adler32-zlib) [ 1278.536994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1278.566143] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 1286.192772] Lustre: DEBUG MARKER: Initial discovery [ 1286.294081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.122@tcp [ 1288.021262] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1288.134758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1288.190840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.122@tcp [ 1288.243932] LNet: There was an unexpected network error while writing to 192.168.206.122: rc = -22 [ 1288.272765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1288.303343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.206.122@tcp -r 1 -m GET -l 3 [ 1288.334350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.206.122@tcp -r 1 -m GET -e local_error [ 1288.366662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.206.122@tcp -r 1 -m PUT -l 6 [ 1288.814424] LNet: 1 local NIs in recovery (showing 1): 192.168.206.22@tcp [ 1289.669216] Lustre: DEBUG MARKER: Force oleg622-client.virtnet to discover 192.168.206.122@tcp (in background) [ 1289.733555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.122@tcp [ 1290.818865] Lustre: DEBUG MARKER: Force oleg622-server to discover 192.168.206.22@tcp [ 1292.777104] LNet: There was an unexpected network error while writing to 192.168.206.122: rc = -22 [ 1292.859692] Lustre: DEBUG MARKER: Wait for 71589 [ 1294.423457] Lustre: DEBUG MARKER: Finished wait on 71589 [ 1294.520577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1294.560262] LNet: There was an unexpected network error while writing to 192.168.206.122: rc = -22 [ 1294.573374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1294.898652] LNet: 72051:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1294.912644] LNetError: 72051:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1294.944667] LNet: Removed LNI 192.168.206.22@tcp [ 1295.367406] Key type .llcrypt unregistered [ 1295.373345] Key type ._llcrypt unregistered [ 1308.558831] Key type ._llcrypt registered [ 1308.561317] Key type .llcrypt registered [ 1309.729872] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 18:44:16 (1786747456) [ 1310.258375] Key type .llcrypt unregistered [ 1310.261563] Key type ._llcrypt unregistered [ 1315.111470] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1315.283559] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1315.295133] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1315.302715] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1317.728846] Key type ._llcrypt registered [ 1317.730461] Key type .llcrypt registered [ 1317.890862] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1317.908932] alg: No test for adler32 (adler32-zlib) [ 1318.846562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1318.915298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1318.941558] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 1319.551061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1319.574591] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1326.710886] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 18:44:33 (1786747473) [ 1326.815095] LNet: 74126:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1326.824364] LNetError: 74126:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1326.830378] LNet: Removed LNI 10.1.2.3@tcp [ 1326.848288] LNet: Removed LNI 192.168.206.22@tcp [ 1327.421841] Key type .llcrypt unregistered [ 1327.423432] Key type ._llcrypt unregistered [ 1332.264686] Key type ._llcrypt registered [ 1332.267399] Key type .llcrypt registered [ 1332.435225] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1332.459904] alg: No test for adler32 (adler32-zlib) [ 1333.362430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1333.409842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1333.428760] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 1333.965310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1333.982419] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1334.793837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.206.22@tcp [ 1335.842356] LNetError: Unexpected error -22 connecting to 192.168.206.22@tcp at host 192.168.206.22:988 [ 1340.143548] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 18:44:47 (1786747487) [ 1340.286812] LNet: 75264:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1340.294338] LNetError: 75264:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1340.299367] LNet: Removed LNI 10.1.2.3@tcp [ 1340.319212] LNet: Removed LNI 192.168.206.22@tcp [ 1340.779655] Key type .llcrypt unregistered [ 1340.783184] Key type ._llcrypt unregistered [ 1345.380463] Key type ._llcrypt registered [ 1345.382746] Key type .llcrypt registered [ 1345.548734] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1345.557906] alg: No test for adler32 (adler32-zlib) [ 1346.426857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1346.468096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1346.483179] LNet: Added LNI 192.168.206.22@tcp1 [8/256/0/180] [ 1347.019473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1347.039314] LNet: Added LNI 192.168.206.22@tcp2 [8/256/0/180] [ 1347.640572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.206.22@tcp1 --nid 192.168.206.22@tcp2 [ 1351.004942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.22@tcp1 192.168.206.22@tcp2 [ 1351.019847] LNetError: 76105:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.206.22@tcp1 to 192.168.206.22@tcp2 - Route table may be misconfigured [ 1351.036516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.22@tcp2 192.168.206.22@tcp1 [ 1355.944994] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 18:45:02 (1786747502) [ 1356.136916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1356.171878] LNet: 76678:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1356.179426] LNetError: 76678:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1357.216235] LNet: Removed LNI 192.168.206.22@tcp1 [ 1357.222315] LNetError: 76678:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1357.241699] LNet: Removed LNI 192.168.206.22@tcp2 [ 1357.268977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1357.322969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1357.357540] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 1358.001536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1358.035966] LNet: Added LNI 192.168.206.22@tcp1 [8/256/0/180] [ 1358.714367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.22@tcp [ 1358.779665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.22@tcp 192.168.206.22@tcp [ 1358.826744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.22@tcp1 192.168.206.22@tcp1 [ 1358.863443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.206.22@tcp -d 192.168.206.22@tcp -e network_timeout [ 1358.892401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.206.22@tcp -d 192.168.206.22@tcp1 -e network_timeout [ 1358.929682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.206.22@tcp1 -d 192.168.206.22@tcp -e network_timeout [ 1358.983209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.206.22@tcp1 -d 192.168.206.22@tcp1 -e network_timeout [ 1359.043279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.22@tcp [ 1359.070073] LNet: There was an unexpected network error while writing to 192.168.206.22: rc = -22 [ 1359.102760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1359.209324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1359.328306] LNet: 1 local NIs in recovery (showing 1): 192.168.206.22@tcp [ 1359.341801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1363.863894] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 18:45:11 (1786747511) [ 1363.911259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1363.923317] LNet: 77533:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1363.931510] LNetError: 77533:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1363.937328] LNet: Removed LNI 192.168.206.22@tcp [ 1364.969701] LNet: Removed LNI 192.168.206.22@tcp1 [ 1364.992535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1365.057638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1365.237367] LNet: 77582:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1365.553677] Key type .llcrypt unregistered [ 1365.557483] Key type ._llcrypt unregistered [ 1374.772747] Key type ._llcrypt registered [ 1374.776137] Key type .llcrypt registered [ 1375.784157] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 18:45:22 (1786747522) [ 1376.035441] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1376.050988] alg: No test for adler32 (adler32-zlib) [ 1376.988219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1378.501540] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1380.136763] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 18:45:27 (1786747527) [ 1380.387414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1380.407676] LNet: 78679:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1380.442128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1380.512676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1380.535803] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 1381.115768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1381.134734] LNet: Added LNI 192.168.206.22@tcp1 [8/256/0/180] [ 1381.816885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.22@tcp [ 1381.857886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.22@tcp1 [ 1381.891646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.22@tcp1 [ 1386.746202] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 18:45:33 (1786747533) [ 1388.045473] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg622-server [ 1389.287364] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 18:45:36 (1786747536) [ 1390.861505] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg622-server [ 1392.037514] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 18:45:39 (1786747539) [ 1393.427826] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg622-server [ 1394.734778] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 18:45:41 (1786747541) [ 1396.084909] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg622-server [ 1397.490528] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 18:45:44 (1786747544) [ 1399.356905] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg622-server [ 1401.027526] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 18:45:47 (1786747547) [ 1402.622475] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg622-server [ 1403.888977] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 18:45:51 (1786747551) [ 1405.516833] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg622-server [ 1407.305433] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 18:45:54 (1786747554) [ 1408.815549] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg622-server [ 1410.221673] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 18:45:57 (1786747557) [ 1411.595212] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg622-server [ 1412.972986] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 18:46:00 (1786747560) [ 1413.035130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1413.057138] LNet: 82549:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1413.067476] LNetError: 82549:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1413.078489] LNet: Removed LNI 192.168.206.22@tcp [ 1413.096145] LNet: Removed LNI 192.168.206.22@tcp1 [ 1413.129564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1413.177503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1413.194587] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 1413.762363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1413.867389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.22@tcp [ 1414.094640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1414.134301] LNet: 82702:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1414.140678] LNetError: 82702:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1414.147369] LNetError: 82702:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1414.180549] LNet: Removed LNI 192.168.206.22@tcp [ 1414.221230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1414.291466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1414.314589] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 1415.038824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.22@tcp [ 1416.304413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1416.324280] LNet: 82857:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1416.331241] LNetError: 82857:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1416.345206] LNet: Removed LNI 192.168.206.22@tcp [ 1416.364154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1416.408656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1416.443263] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 1417.215422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1417.305943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.22@tcp [ 1417.475600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1418.551274] LNet: Removed LNI 192.168.206.22@tcp [ 1418.586433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1418.664585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1418.680023] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 1419.305075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.22@tcp [ 1419.515071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1419.543429] LNetError: 83149:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1419.556930] LNetError: 83149:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1419.567785] LNet: Removed LNI 192.168.206.22@tcp [ 1419.606356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1419.672229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1419.700461] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 1420.295339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1420.398920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.22@tcp [ 1420.698509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1420.746186] LNet: 83296:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1420.754116] LNet: 83296:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1421.815951] LNet: Removed LNI 192.168.206.22@tcp [ 1421.859310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1421.930645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1421.959253] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 1422.690565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.22@tcp [ 1423.010034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1424.103469] LNet: Removed LNI 192.168.206.22@tcp [ 1424.130364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1424.185143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1424.215106] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 1424.848415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1424.978742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.22@tcp [ 1426.327407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1426.352234] LNetError: 83598:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1426.359760] LNetError: 83598:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1426.381697] LNet: Removed LNI 192.168.206.22@tcp [ 1426.396967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1426.467097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1426.494231] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 1427.551318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.22@tcp [ 1427.789887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1428.839927] LNet: Removed LNI 192.168.206.22@tcp [ 1428.879611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1428.930939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1428.981702] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 1429.727362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1429.798929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.22@tcp [ 1430.060596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1430.089931] LNet: 83890:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1430.096971] LNet: 83890:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1431.143311] LNet: Removed LNI 192.168.206.22@tcp [ 1431.181424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1431.326947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1431.402976] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 1432.264383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.22@tcp [ 1432.582809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1433.643470] LNet: Removed LNI 192.168.206.22@tcp [ 1433.679493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1433.752342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1433.780019] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 1434.401324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1434.473639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.22@tcp [ 1434.691459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1434.709474] LNetError: 84183:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1434.714793] LNetError: 84183:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1435.764175] LNet: Removed LNI 192.168.206.22@tcp [ 1435.800464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1435.879735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1435.905584] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 1436.562647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.22@tcp [ 1436.863164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1437.926973] LNet: Removed LNI 192.168.206.22@tcp [ 1437.963715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1438.043750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1438.070226] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 1438.750930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1438.821662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1438.850747] LNet: Removed LNI 192.168.206.22@tcp [ 1438.875939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1438.951912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1438.970803] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 1439.584146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1439.666950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1439.694147] LNet: Removed LNI 192.168.206.22@tcp [ 1439.710959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1439.759886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1439.845936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1439.889521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1439.935776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1440.023220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1440.074450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1440.133758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1440.221644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1440.257131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1440.321735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1440.343415] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 1440.929173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1440.960111] LNet: Removed LNI 192.168.206.22@tcp [ 1440.972518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1441.022479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1446.307994] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 18:46:33 (1786747593) [ 1446.392322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1446.421555] LNet: 85343:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1446.430319] LNet: 85343:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 1446.465247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1446.517849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1446.550432] LNet: Added LNI 192.168.206.22@tcp231 [8/256/0/180] [ 1447.588817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1447.611149] LNet: Removed LNI 192.168.206.22@tcp231 [ 1447.631541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1447.681924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1447.704589] LNet: Added LNI 192.168.206.22@tcp231 [8/256/0/180] [ 1447.710650] LNet: 85496:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1448.009620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1448.034726] LNet: Removed LNI 192.168.206.22@tcp231 [ 1448.047913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1448.077185] LNet: Added LNI 192.168.206.22@tcp231 [8/256/0/180] [ 1449.060477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1449.080608] LNet: Removed LNI 192.168.206.22@tcp231 [ 1449.094701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1449.145050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1449.174800] LNet: Added LNI 192.168.206.22@tcp231 [8/256/0/0] [ 1449.179708] LNet: 85648:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1449.547281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1449.576503] LNet: Removed LNI 192.168.206.22@tcp231 [ 1449.594370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1449.614158] LNet: Added LNI 192.168.206.22@tcp231 [8/256/0/0] [ 1450.503601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1450.539753] LNet: Removed LNI 192.168.206.22@tcp231 [ 1450.558424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1450.639309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1450.690728] LNet: Added LNI 192.168.206.22@tcp231 [8/256/0/60] [ 1450.697581] LNet: 85799:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1451.217949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1451.248772] LNet: Removed LNI 192.168.206.22@tcp231 [ 1451.275201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1451.305592] LNet: Added LNI 192.168.206.22@tcp231 [8/256/0/60] [ 1452.220510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1452.242096] LNetError: 85943:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1452.246729] LNetError: 85943:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 1452.259324] LNet: Removed LNI 192.168.206.22@tcp231 [ 1452.278652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1452.322941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1452.343256] LNet: Added LNI 192.168.206.22@tcp231 [8/256/0/180] [ 1452.669696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1452.698586] LNet: Removed LNI 192.168.206.22@tcp231 [ 1452.717425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1452.744713] LNet: Added LNI 192.168.206.22@tcp231 [8/256/0/180] [ 1458.308512] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 18:46:45 (1786747605) [ 1458.442780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1458.479791] LNet: Removed LNI 192.168.206.22@tcp231 [ 1458.503248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1458.563945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1458.579353] LNet: Added LNI 192.168.206.22@tcp232 [8/256/0/180] [ 1459.042529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1459.071951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1459.108260] LNet: Removed LNI 192.168.206.22@tcp232 [ 1459.123836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1459.143423] LNet: Added LNI 192.168.206.22@tcp232 [8/256/0/180] [ 1459.700693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1459.747257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1459.782046] LNet: Removed LNI 192.168.206.22@tcp232 [ 1459.802325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1459.825323] LNet: Added LNI 192.168.206.22@tcp232 [8/256/0/180] [ 1460.448670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1460.484643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1460.524481] LNet: Removed LNI 192.168.206.22@tcp232 [ 1460.555187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1460.581151] LNet: Added LNI 192.168.206.22@tcp232 [8/256/0/180] [ 1461.176811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1461.217461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1461.248688] LNet: Removed LNI 192.168.206.22@tcp232 [ 1461.260714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1466.022909] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 18:46:53 (1786747613) [ 1466.945827] Key type .llcrypt unregistered [ 1466.949798] Key type ._llcrypt unregistered [ 1471.576882] Key type ._llcrypt registered [ 1471.580045] Key type .llcrypt registered [ 1471.754360] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1471.768262] alg: No test for adler32 (adler32-zlib) [ 1472.633880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1472.668594] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 1478.234421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.122@tcp [ 1484.776976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1484.832130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1484.852336] LNet: Added LNI 192.168.206.22@tcp1 [8/256/0/180] [ 1485.922663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.22@tcp -d 192.168.206.122@tcp -m GET -e local_interrupt -r 2 [ 1485.957214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.22@tcp -d 192.168.206.122@tcp1 -m GET -e local_interrupt -r 2 [ 1485.995570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.22@tcp -d 192.168.206.22@tcp -m GET -e local_interrupt -r 2 [ 1486.031757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.22@tcp -d 192.168.206.22@tcp1 -m GET -e local_interrupt -r 2 [ 1486.071894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.22@tcp1 -d 192.168.206.122@tcp -m GET -e local_interrupt -r 2 [ 1486.099751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.22@tcp1 -d 192.168.206.122@tcp1 -m GET -e local_interrupt -r 2 [ 1486.134538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.22@tcp1 -d 192.168.206.22@tcp -m GET -e local_interrupt -r 2 [ 1486.168118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.22@tcp1 -d 192.168.206.22@tcp1 -m GET -e local_interrupt -r 2 [ 1486.201089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.22@tcp 192.168.206.122@tcp [ 1486.227179] LNet: There was an unexpected network error while writing to 192.168.206.122: rc = -22 [ 1486.233802] LNet: 1 local NIs in recovery (showing 1): 192.168.206.22@tcp [ 1486.439555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1486.461760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1486.485529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1486.508820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1486.532477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1486.554604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1486.579526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1486.601630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1486.612428] LNet: 88453:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1486.627600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1486.654745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1486.681523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1486.704415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1486.735720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1486.768666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1487.170626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.22@tcp -d 192.168.206.122@tcp -m GET -e local_dropped -r 2 [ 1487.199989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.22@tcp -d 192.168.206.122@tcp1 -m GET -e local_dropped -r 2 [ 1487.224852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.22@tcp -d 192.168.206.22@tcp -m GET -e local_dropped -r 2 [ 1487.270183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.22@tcp -d 192.168.206.22@tcp1 -m GET -e local_dropped -r 2 [ 1487.315959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.22@tcp1 -d 192.168.206.122@tcp -m GET -e local_dropped -r 2 [ 1487.348219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.22@tcp1 -d 192.168.206.122@tcp1 -m GET -e local_dropped -r 2 [ 1487.377990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.22@tcp1 -d 192.168.206.22@tcp -m GET -e local_dropped -r 2 [ 1487.415143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.22@tcp1 -d 192.168.206.22@tcp1 -m GET -e local_dropped -r 2 [ 1487.449669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.22@tcp 192.168.206.122@tcp [ 1487.470338] LNet: There was an unexpected network error while writing to 192.168.206.122: rc = -22 [ 1487.648176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1487.677267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1487.702906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1487.732517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1487.757866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1487.792990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1487.819043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1487.846503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1487.859424] LNet: 88573:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1487.882736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1487.910192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1487.933238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1487.961507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1488.001531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1488.040612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1488.483444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.22@tcp -d 192.168.206.122@tcp -m GET -e local_aborted -r 2 [ 1488.512512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.22@tcp -d 192.168.206.122@tcp1 -m GET -e local_aborted -r 2 [ 1488.543433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.22@tcp -d 192.168.206.22@tcp -m GET -e local_aborted -r 2 [ 1488.573670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.22@tcp -d 192.168.206.22@tcp1 -m GET -e local_aborted -r 2 [ 1488.599621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.22@tcp1 -d 192.168.206.122@tcp -m GET -e local_aborted -r 2 [ 1488.628403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.22@tcp1 -d 192.168.206.122@tcp1 -m GET -e local_aborted -r 2 [ 1488.665421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.22@tcp1 -d 192.168.206.22@tcp -m GET -e local_aborted -r 2 [ 1488.710068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.22@tcp1 -d 192.168.206.22@tcp1 -m GET -e local_aborted -r 2 [ 1488.747366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.22@tcp 192.168.206.122@tcp [ 1488.765277] LNet: There was an unexpected network error while writing to 192.168.206.122: rc = -22 [ 1489.007591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1489.048815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1489.080879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1489.109682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1489.133694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1489.160642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1489.190916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1489.224325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1489.241043] LNet: 88693:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1489.264742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1489.300783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1489.327937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1489.355246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1489.381318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1489.406762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1489.829743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.22@tcp -d 192.168.206.122@tcp -m GET -e local_no_route -r 2 [ 1489.877207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.22@tcp -d 192.168.206.122@tcp1 -m GET -e local_no_route -r 2 [ 1489.927078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.22@tcp -d 192.168.206.22@tcp -m GET -e local_no_route -r 2 [ 1489.962605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.22@tcp -d 192.168.206.22@tcp1 -m GET -e local_no_route -r 2 [ 1489.991497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.22@tcp1 -d 192.168.206.122@tcp -m GET -e local_no_route -r 2 [ 1490.017914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.22@tcp1 -d 192.168.206.122@tcp1 -m GET -e local_no_route -r 2 [ 1490.049743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.22@tcp1 -d 192.168.206.22@tcp -m GET -e local_no_route -r 2 [ 1490.089388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.22@tcp1 -d 192.168.206.22@tcp1 -m GET -e local_no_route -r 2 [ 1490.124771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.22@tcp 192.168.206.122@tcp [ 1490.293401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1490.320725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1490.359808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1490.391992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1490.426367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1490.456957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1490.495285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1490.530694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1490.562729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1490.586199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1490.615989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1490.641640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1490.663509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1490.685780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1491.127245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.22@tcp -d 192.168.206.122@tcp -m GET -e local_timeout -r 2 [ 1491.154778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.22@tcp -d 192.168.206.122@tcp1 -m GET -e local_timeout -r 2 [ 1491.187675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.22@tcp -d 192.168.206.22@tcp -m GET -e local_timeout -r 2 [ 1491.236883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.22@tcp -d 192.168.206.22@tcp1 -m GET -e local_timeout -r 2 [ 1491.279707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.22@tcp1 -d 192.168.206.122@tcp -m GET -e local_timeout -r 2 [ 1491.319864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.22@tcp1 -d 192.168.206.122@tcp1 -m GET -e local_timeout -r 2 [ 1491.377663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.22@tcp1 -d 192.168.206.22@tcp -m GET -e local_timeout -r 2 [ 1491.420839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.22@tcp1 -d 192.168.206.22@tcp1 -m GET -e local_timeout -r 2 [ 1491.462903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.22@tcp 192.168.206.122@tcp [ 1491.486248] LNet: There was an unexpected network error while writing to 192.168.206.122: rc = -22 [ 1491.490482] LNet: Skipped 1 previous similar message [ 1491.685383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1491.710685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1491.752284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1491.783261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1491.829086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1491.861468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1491.888171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1491.914881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1491.930106] LNet: 88933:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1491.940044] LNet: 88933:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1491.958353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1491.985710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1492.014537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1492.037839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1492.063679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1492.096713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1492.461946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.22@tcp -d 192.168.206.122@tcp -m GET -e remote_dropped -r 2 [ 1492.487957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.22@tcp -d 192.168.206.122@tcp1 -m GET -e remote_dropped -r 2 [ 1492.516433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.22@tcp -d 192.168.206.22@tcp -m GET -e remote_dropped -r 2 [ 1492.543872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.22@tcp -d 192.168.206.22@tcp1 -m GET -e remote_dropped -r 2 [ 1492.570988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.22@tcp1 -d 192.168.206.122@tcp -m GET -e remote_dropped -r 2 [ 1492.597692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.22@tcp1 -d 192.168.206.122@tcp1 -m GET -e remote_dropped -r 2 [ 1492.630654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.22@tcp1 -d 192.168.206.22@tcp -m GET -e remote_dropped -r 2 [ 1492.662346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.22@tcp1 -d 192.168.206.22@tcp1 -m GET -e remote_dropped -r 2 [ 1492.700617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.22@tcp 192.168.206.122@tcp [ 1492.726870] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.122@tcp [ 1492.911943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1492.938223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1492.963552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1492.988626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1493.019912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1493.070706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1493.112707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1493.140910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1493.162442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1493.186119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1493.212788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1493.257655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1493.297812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1493.326659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1496.201424] LNet: 89131:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1496.208544] LNetError: 89131:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1496.219424] LNet: Removed LNI 192.168.206.22@tcp [ 1496.232336] LNet: Removed LNI 192.168.206.22@tcp1 [ 1496.532626] Key type .llcrypt unregistered [ 1496.536849] Key type ._llcrypt unregistered [ 1509.225769] Key type ._llcrypt registered [ 1509.237647] Key type .llcrypt registered [ 1510.589205] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 18:47:37 (1786747657) [ 1510.973655] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1510.995168] alg: No test for adler32 (adler32-zlib) [ 1511.905681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1511.944991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1511.968849] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 1512.793688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1512.848816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.22@tcp [ 1512.877023] LNetError: 90108:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.206.22@tcp to recovery [ 1512.928755] LNet: 1 local NIs in recovery (showing 1): 192.168.206.22@tcp [ 1512.996646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.206.22@tcp [ 1513.130680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1519.247413] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 18:47:46 (1786747666) [ 1519.399545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1519.458078] LNet: 90698:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1519.470136] LNetError: 90698:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1519.493459] LNet: Removed LNI 192.168.206.22@tcp [ 1519.516443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1519.560634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1519.611337] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 1520.443341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1520.463281] LNet: Added LNI 192.168.206.22@tcp1 [8/256/0/180] [ 1521.395810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.206.22@tcp [ 1521.414479] LNetError: 90965:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.206.22@tcp to recovery [ 1521.573642] LNet: There was an unexpected network error while writing to 192.168.206.22: rc = -22 [ 1522.685955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.206.22@tcp [ 1529.938942] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 18:47:56 (1786747676) [ 1531.765919] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg622-server [ 1533.086577] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 18:48:00 (1786747680) [ 1534.725277] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg622-server [ 1536.591509] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 18:48:03 (1786747683) [ 1538.127497] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg622-server [ 1539.547962] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 18:48:06 (1786747686) [ 1539.767834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1539.815192] LNet: 92586:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1539.823267] LNetError: 92586:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1539.830751] LNet: Removed LNI 192.168.206.22@tcp [ 1539.861856] LNet: Removed LNI 192.168.206.22@tcp1 [ 1539.889806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1539.939974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1539.970743] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 1540.669685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1540.732425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1540.784388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1540.847934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1540.899449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1540.935363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1546.019367] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 18:48:13 (1786747693) [ 1546.109361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1546.126028] LNet: 93303:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1546.132194] LNetError: 93303:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1546.137728] LNetError: 93303:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1546.156459] LNet: Removed LNI 192.168.206.22@tcp [ 1546.169160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1546.216934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1546.243632] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 1551.666250] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 18:48:18 (1786747698) [ 1551.793554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1551.830224] LNet: 94011:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1551.839233] LNetError: 94011:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1551.863199] LNet: Removed LNI 192.168.206.22@tcp [ 1551.881624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1551.932234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1551.954353] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 1557.338808] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 18:48:24 (1786747704) [ 1558.476513] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1560.003872] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 18:48:26 (1786747706) [ 1560.492046] LNet: 95049:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1560.504023] LNetError: 95049:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1560.529875] LNet: Removed LNI 192.168.206.22@tcp [ 1561.010742] Key type .llcrypt unregistered [ 1561.014130] Key type ._llcrypt unregistered [ 1566.300639] Key type ._llcrypt registered [ 1566.302977] Key type .llcrypt registered [ 1566.448026] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1566.463912] alg: No test for adler32 (adler32-zlib) [ 1567.338354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1567.380570] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 1573.533143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.122@tcp [ 1578.367039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.206.122@tcp [ 1579.742248] LNet: 95474:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1579.755154] LNetError: 95474:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1579.783103] LNet: Removed LNI 192.168.206.22@tcp [ 1580.152840] Key type .llcrypt unregistered [ 1580.158709] Key type ._llcrypt unregistered [ 1589.976353] Key type ._llcrypt registered [ 1589.982676] Key type .llcrypt registered [ 1591.498672] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1592.877846] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1594.302621] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 18:49:01 (1786747741) [ 1595.554852] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1596.871583] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 18:49:04 (1786747744) [ 1598.452789] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg622-server [ 1600.037643] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 18:49:06 (1786747746) [ 1601.500864] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg622-server [ 1602.777885] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 18:49:09 (1786747749) [ 1604.516491] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1604.587594] alg: No test for adler32 (adler32-zlib) [ 1605.624715] Lustre: Lustre: Build Version: 2.17.57_3_g6085841 [ 1605.815343] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 1607.504169] Key type lgssc registered [ 1608.624542] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1624.861789] Key type lgssc unregistered [ 1625.154919] LNet: 98651:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1625.162184] LNetError: 98651:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1625.178776] LNet: Removed LNI 192.168.206.22@tcp [ 1625.808467] Key type .llcrypt unregistered [ 1625.812658] Key type ._llcrypt unregistered [ 1636.856370] Key type ._llcrypt registered [ 1636.858698] Key type .llcrypt registered [ 1637.813437] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 18:49:44 (1786747784) [ 1638.116862] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1638.128769] alg: No test for adler32 (adler32-zlib) [ 1639.000926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1639.060851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1639.107661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1639.138596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1639.173570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1644.730815] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 18:49:51 (1786747791) [ 1644.932730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1644.959273] LNet: 100066:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1644.980382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1645.026779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1645.049696] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 1645.736275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1645.762827] LNet: 100210:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1645.768655] LNetError: 100210:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1645.785383] LNet: Removed LNI 192.168.206.22@tcp [ 1645.816202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1645.879671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1645.913568] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 1646.252502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1646.276818] LNetError: 100238:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1646.291905] LNet: Removed LNI 192.168.206.22@tcp [ 1646.312854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1646.366177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1646.387802] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 1651.667324] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 18:49:58 (1786747798) [ 1651.796472] LNet: 100825:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1651.800492] LNet: 100825:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1651.807725] LNetError: 100825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1651.833273] LNet: Removed LNI 192.168.206.22@tcp [ 1652.390412] Key type .llcrypt unregistered [ 1652.396082] Key type ._llcrypt unregistered [ 1657.991573] Key type ._llcrypt registered [ 1657.995094] Key type .llcrypt registered [ 1658.152503] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1658.167736] alg: No test for adler32 (adler32-zlib) [ 1659.096142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1659.154111] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 1659.157402] LNet: *** cfs_fail_loc=e003, val=100*** [ 1659.818025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.22@tcp [ 1659.834107] LNet: *** cfs_fail_loc=e003, val=100*** [ 1659.837473] LNet: Skipped 2 previous similar messages [ 1659.843825] Ping reply received with mapping: [ 1659.845617] found metadata reply NI[192.168.206.22@tcp] [ 1659.847585] 16 ce a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1659.849931] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1659.851978] 5a 5a 5a 5a 5a 5a 5a 5a [ 1659.968125] LNet: 101229:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1659.978152] LNetError: 101229:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1660.000983] LNet: Removed LNI 192.168.206.22@tcp [ 1660.563537] Key type .llcrypt unregistered [ 1660.566986] Key type ._llcrypt unregistered [ 1670.041568] Key type ._llcrypt registered [ 1670.049517] Key type .llcrypt registered [ 1671.230931] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 18:50:18 (1786747818) [ 1671.684341] Key type .llcrypt unregistered [ 1671.686053] Key type ._llcrypt unregistered [ 1677.595680] Key type ._llcrypt registered [ 1677.598748] Key type .llcrypt registered [ 1677.770226] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1677.784221] alg: No test for adler32 (adler32-zlib) [ 1678.913574] LNetError: 102287:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1679.232560] Key type .llcrypt unregistered [ 1679.236652] Key type ._llcrypt unregistered [ 1683.611167] Key type ._llcrypt registered [ 1683.613566] Key type .llcrypt registered [ 1684.853949] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 18:50:31 (1786747831) [ 1685.410390] Key type .llcrypt unregistered [ 1685.417599] Key type ._llcrypt unregistered [ 1690.129573] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1690.139727] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1692.656744] Key type ._llcrypt registered [ 1692.662224] Key type .llcrypt registered [ 1692.884526] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1692.898396] alg: No test for adler32 (adler32-zlib) [ 1693.815835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1693.892574] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1702.015414] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 18:50:49 (1786747849) [ 1702.163654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1702.206624] LNet: 104070:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1702.217777] LNetError: 104070:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1702.265583] LNet: Removed LNI 10.1.2.3@tcp [ 1702.304120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1703.063749] Key type .llcrypt unregistered [ 1703.070342] Key type ._llcrypt unregistered [ 1709.172364] Key type ._llcrypt registered [ 1709.174297] Key type .llcrypt registered [ 1709.296176] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1709.309711] alg: No test for adler32 (adler32-zlib) [ 1710.178819] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 1710.261790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.22@tcp [ 1710.456665] LNet: 104367:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1710.469161] LNetError: 104367:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1710.499118] LNet: Removed LNI 192.168.206.22@tcp [ 1711.282634] Key type .llcrypt unregistered [ 1711.284299] Key type ._llcrypt unregistered [ 1722.961784] Key type ._llcrypt registered [ 1722.964806] Key type .llcrypt registered [ 1724.386761] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 18:51:10 (1786747870) [ 1725.074631] Key type .llcrypt unregistered [ 1725.079389] Key type ._llcrypt unregistered [ 1732.757646] Key type ._llcrypt registered [ 1732.760720] Key type .llcrypt registered [ 1732.985590] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1733.006561] alg: No test for adler32 (adler32-zlib) [ 1740.004082] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 18:51:27 (1786747887) [ 1740.236937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1740.286495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1740.329494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1740.347700] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 1740.956523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1740.998718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1741.023828] LNet: Removed LNI 192.168.206.22@tcp [ 1741.261242] LNet: 106122:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1741.608912] Key type .llcrypt unregistered [ 1741.612702] Key type ._llcrypt unregistered [ 1751.059240] Key type ._llcrypt registered [ 1751.061590] Key type .llcrypt registered [ 1752.142931] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 18:51:39 (1786747899) [ 1752.415740] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1752.433307] alg: No test for adler32 (adler32-zlib) [ 1753.344490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1753.402318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1753.441179] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 1754.052909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.22@tcp [ 1754.090908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.206.22@tcp [ 1754.110647] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1754.115870] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1754.119680] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1754, 0, 0 [ 1754.124039] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1754.126597] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1754.130468] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1754.134116] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1754, 0, 0 [ 1754.139096] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1754.141971] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1754.145772] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1754.147885] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1754, 0, 1804 [ 1754.153056] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1754.155756] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1754.159277] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1754.161792] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1754, 0, 1804 [ 1754.166538] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1759.426857] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 18:51:46 (1786747906) [ 1759.802223] LNet: 107662:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1759.812429] LNetError: 107662:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1759.823417] LNet: Removed LNI 192.168.206.22@tcp [ 1760.321411] Key type .llcrypt unregistered [ 1760.330953] Key type ._llcrypt unregistered [ 1764.890632] Key type ._llcrypt registered [ 1764.892489] Key type .llcrypt registered [ 1765.054225] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1765.065648] alg: No test for adler32 (adler32-zlib) [ 1765.941929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1765.963938] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 1772.094479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.122@tcp [ 1780.702720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1780.751453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1780.784351] LNet: Added LNI 192.168.206.22@tcp1 [8/256/0/180] [ 1781.649170] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1781.817277] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1781.821443] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1781.825073] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1784.158192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1784.186619] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1793.844318] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 18:52:20 (1786747940) [ 1793.971336] LNet: 109166:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1793.976435] LNetError: 109166:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1793.993101] LNet: Removed LNI 192.168.206.22@tcp [ 1793.995504] LNet: Removed LNI 192.168.206.22@tcp1 [ 1794.011960] LNet: Removed LNI 10.1.2.3@tcp99 [ 1794.593951] Key type .llcrypt unregistered [ 1794.599300] Key type ._llcrypt unregistered [ 1799.788744] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1799.796856] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1802.360445] Key type ._llcrypt registered [ 1802.362587] Key type .llcrypt registered [ 1802.539684] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1802.553832] alg: No test for adler32 (adler32-zlib) [ 1803.493821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1803.557714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1803.584454] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 1804.255137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1804.277221] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1806.310732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 1806.365048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1806.431185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 1806.460927] LNetError: 109921:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1806.502626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1806.540780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.22@tcp [ 1806.821928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1806.878759] LNet: 109941:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1806.884678] LNetError: 109941:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1806.891942] LNet: Removed LNI 10.1.2.3@tcp [ 1807.911863] LNet: Removed LNI 192.168.206.22@tcp [ 1808.474106] Key type .llcrypt unregistered [ 1808.485090] Key type ._llcrypt unregistered [ 1818.975461] Key type ._llcrypt registered [ 1818.979743] Key type .llcrypt registered [ 1820.220691] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 18:52:47 (1786747967) [ 1820.660000] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1820.681229] alg: No test for adler32 (adler32-zlib) [ 1821.603144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1821.673634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1821.714482] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 1822.361938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.22@tcp [ 1822.514338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg622-client@tcp [ 1828.014582] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 18:52:54 (1786747974) [ 1828.109238] LNet: 111470:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1828.115769] LNetError: 111470:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1828.135784] LNet: Removed LNI 192.168.206.22@tcp [ 1828.754377] Key type .llcrypt unregistered [ 1828.756275] Key type ._llcrypt unregistered [ 1833.489517] Key type ._llcrypt registered [ 1833.492575] Key type .llcrypt registered [ 1833.612309] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1833.621966] alg: No test for adler32 (adler32-zlib) [ 1834.478405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 1834.496457] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 1834.522807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1839.418877] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 18:53:06 (1786747986) [ 1840.510335] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 1842.104252] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 18:53:08 (1786747988) [ 1843.298906] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 1844.715710] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 18:53:11 (1786747991) [ 1844.810552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1844.847439] LNet: 112892:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1844.859240] LNetError: 112892:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1844.882384] LNet: Removed LNI 192.168.206.22@tcp [ 1844.909673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1844.999750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1845.024522] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 1845.731794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1845.800872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1851.842753] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 18:53:18 (1786747998) [ 1851.951693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1851.974398] LNet: 113604:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1851.986930] LNetError: 113604:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1852.036618] LNet: Removed LNI 192.168.206.22@tcp [ 1852.094838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1852.175766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 1852.220627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1852.887491] Key type .llcrypt unregistered [ 1852.889761] Key type ._llcrypt unregistered [ 1863.108520] Key type ._llcrypt registered [ 1863.115350] Key type .llcrypt registered [ 1864.353473] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 18:53:31 (1786748011) [ 1864.678538] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1864.690625] alg: No test for adler32 (adler32-zlib) [ 1865.540201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1865.581784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1865.616552] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 1866.268327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1866.345788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.22@tcp [ 1872.161834] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 18:53:39 (1786748019) [ 1872.231634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1872.265866] LNet: 115160:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1872.271425] LNetError: 115160:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1872.290491] LNet: Removed LNI 192.168.206.22@tcp [ 1872.307504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1872.369955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 1872.404248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 1878.467884] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 18:53:45 (1786748025) [ 1878.732139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1878.759172] LNet: 115747:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1878.792734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1878.875873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1879.222463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1879.247541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1879.297607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 1879.561912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1879.592728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1879.681221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1879.724953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1879.807273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1880.172017] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1880.222758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1880.299810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 1880.688128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1880.738757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1885.774857] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 18:53:52 (1786748032) [ 1886.018514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1886.054513] LNet: 116380:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1886.061564] LNet: 116380:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1886.117457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1886.209091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1886.240038] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 1886.994681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1887.017471] LNet: Added LNI 192.168.206.22@tcp2 [8/256/0/180] [ 1887.863707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1887.910569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.22@tcp [ 1888.073887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1888.101386] LNet: 116659:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1888.107393] LNetError: 116659:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1888.113363] LNet: Removed LNI 192.168.206.22@tcp [ 1888.124247] LNet: Removed LNI 192.168.206.22@tcp2 [ 1888.144821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1888.204143] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 1888.207581] LNet: Added LNI 192.168.206.22@tcp2 [8/256/0/180] [ 1888.211258] LNet: 116665:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1888.632914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.22@tcp [ 1888.828255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1889.888260] LNet: Removed LNI 192.168.206.22@tcp [ 1889.891818] LNetError: 116702:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1889.907633] LNetError: 116702:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1889.934442] LNet: Removed LNI 192.168.206.22@tcp2 [ 1889.958357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1890.059783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1890.090888] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 1890.777282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1890.812846] LNet: Added LNI 192.168.206.22@tcp2 [8/256/0/180] [ 1891.563502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.22@tcp [ 1891.630064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1891.728502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.22@tcp [ 1891.938881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1891.955836] LNetError: 116987:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1892.962836] LNet: Removed LNI 192.168.206.22@tcp [ 1894.001394] LNet: Removed LNI 192.168.206.22@tcp2 [ 1894.033871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1894.076069] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 1894.081770] LNet: Added LNI 192.168.206.22@tcp2 [8/256/0/180] [ 1894.085842] LNet: 116993:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1894.610229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.22@tcp [ 1894.772839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1894.789257] LNet: 117030:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1894.794632] LNet: 117030:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1894.800068] LNet: Removed LNI 192.168.206.22@tcp [ 1895.857494] LNet: Removed LNI 192.168.206.22@tcp2 [ 1895.891108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1895.962495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1895.978252] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 1896.581830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1896.596854] LNet: Added LNI 192.168.206.22@tcp2 [8/256/0/180] [ 1897.257484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1897.295922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.22@tcp [ 1897.489477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1897.505163] LNetError: 117309:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1897.513496] LNetError: 117309:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1898.529481] LNet: Removed LNI 192.168.206.22@tcp [ 1898.542883] LNet: Removed LNI 192.168.206.22@tcp2 [ 1898.588923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1898.646604] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 1898.652874] LNet: Added LNI 192.168.206.22@tcp2 [8/256/0/180] [ 1898.657909] LNet: 117316:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1899.103972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.22@tcp [ 1899.274163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1900.320747] LNet: Removed LNI 192.168.206.22@tcp [ 1900.331400] LNet: Removed LNI 192.168.206.22@tcp2 [ 1900.378200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1900.426835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1900.446699] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 1901.063764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1901.089938] LNet: Added LNI 192.168.206.22@tcp2 [8/256/0/180] [ 1901.796305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.22@tcp [ 1901.828797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1901.880389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.22@tcp [ 1902.095358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1903.136793] LNet: Removed LNI 192.168.206.22@tcp [ 1904.168208] LNet: Removed LNI 192.168.206.22@tcp2 [ 1904.211668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1904.278583] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 1904.283352] LNet: Added LNI 192.168.206.22@tcp2 [8/256/0/180] [ 1904.288128] LNet: 117643:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1904.758334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.22@tcp [ 1904.945839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1904.982236] LNet: 117680:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1904.991100] LNet: 117680:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1905.002894] LNet: Removed LNI 192.168.206.22@tcp [ 1906.028065] LNet: Removed LNI 192.168.206.22@tcp2 [ 1906.050851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1906.111257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1906.133827] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 1906.678332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1906.693342] LNet: Added LNI 192.168.206.22@tcp2 [8/256/0/180] [ 1907.439500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1907.479039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1907.535984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.22@tcp [ 1907.825974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1907.851887] LNetError: 117977:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1907.857042] LNetError: 117977:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 1907.868150] LNet: Removed LNI 192.168.206.22@tcp [ 1907.879036] LNet: Removed LNI 192.168.206.22@tcp2 [ 1907.894095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1907.935630] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 1907.944307] LNet: Added LNI 192.168.206.22@tcp2 [8/256/0/180] [ 1908.281706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.22@tcp [ 1908.566404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1909.601204] LNet: Removed LNI 192.168.206.22@tcp [ 1909.612552] LNet: Removed LNI 192.168.206.22@tcp2 [ 1909.632602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1909.695066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1909.718892] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 1910.269572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1910.285990] LNet: Added LNI 192.168.206.22@tcp2 [8/256/0/180] [ 1910.968502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.22@tcp [ 1911.006696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1911.036826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1911.111058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.22@tcp [ 1911.395595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1912.480310] LNet: Removed LNI 192.168.206.22@tcp [ 1913.514732] LNet: Removed LNI 192.168.206.22@tcp2 [ 1913.543274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1913.625867] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 1913.634319] LNet: Added LNI 192.168.206.22@tcp2 [8/256/0/180] [ 1913.638363] LNet: 118343:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1913.644533] LNet: 118343:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1914.138403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.22@tcp [ 1921.077318] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 1922.493819] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 18:54:29 (1786748069) [ 1922.636536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1922.681894] LNet: 119098:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1922.691866] LNet: 119098:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1923.744184] LNet: Removed LNI 192.168.206.22@tcp [ 1924.795603] LNet: Removed LNI 192.168.206.22@tcp2 [ 1924.831750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1925.167488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1925.189204] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 1925.911476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1925.935536] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1926.956350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1987.363595] LNet: 1 local NIs in recovery (showing 1): 10.1.2.3@tcp [ 1987.425883] LNet: 119376:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1987.439935] LNetError: 119376:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1987.448778] LNetError: 119376:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2058.145028] LNet: Removed LNI 10.1.2.3@tcp [ 2058.151730] LNetError: 119376:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2058.176209] LNet: Removed LNI 192.168.206.22@tcp [ 2058.217546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2058.336514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2058.372472] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 2058.381919] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2060.436588] Lustre: DEBUG MARKER: sanity-lnet test_450: @@@@@@ FAIL: test_450 failed with 1 [ 2072.061191] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 18:56:58 (1786748218) [ 2072.221507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2072.247105] LNet: 120126:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2072.255896] LNet: Removed LNI 10.1.2.3@tcp [ 2072.277334] LNet: Removed LNI 192.168.206.22@tcp [ 2072.322922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2072.395467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2072.429038] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 2073.437797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2073.489913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.22@tcp [ 2073.560240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.22@tcp [ 2073.619881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.22@tcp [ 2073.667701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.22@tcp [ 2073.715430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.22@tcp [ 2073.796860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.22@tcp [ 2073.857631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.22@tcp [ 2073.910704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.22@tcp [ 2073.945953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.22@tcp [ 2073.986541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.22@tcp [ 2074.183172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2082.375496] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 18:57:09 (1786748229) [ 2082.772793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2083.828648] LNet: Removed LNI 192.168.206.22@tcp [ 2083.893066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2084.461467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2084.500048] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 2085.507580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2085.531945] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2086.378875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.22@tcp [ 2086.474890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2086.580368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.206.22@tcp -m PUT -e local_timeout -r 1 [ 2086.694044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2086.763829] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2086.767200] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2087.006053] LNetError: 120901:0:(lib-move.c:4801:lnet_parse()) 192.168.206.22@tcp, src 192.168.206.22@tcp: Dropping PUT (error -108 looking up sender) [ 2088.035227] LNet: Removed LNI 10.1.2.3@tcp [ 2089.066819] LNet: Removed LNI 192.168.206.22@tcp [ 2089.532565] Key type .llcrypt unregistered [ 2089.534337] Key type ._llcrypt unregistered [ 2100.338946] Key type ._llcrypt registered [ 2100.342976] Key type .llcrypt registered [ 2101.510405] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 18:57:28 (1786748248) [ 2101.885592] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2101.911593] alg: No test for adler32 (adler32-zlib) [ 2102.818563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2103.280613] LNet: 122043:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2104.020351] Key type .llcrypt unregistered [ 2104.023512] Key type ._llcrypt unregistered [ 2109.074404] Key type ._llcrypt registered [ 2109.076972] Key type .llcrypt registered [ 2109.211889] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2109.234468] alg: No test for adler32 (adler32-zlib) [ 2110.105403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2110.134730] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 2116.301410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.122@tcp [ 2116.441650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2116.908102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.206.122@tcp [ 2117.149149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2117.178132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2117.227481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2117.251794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2117.275754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2117.307267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2117.345148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2117.345529] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.122@tcp [ 2117.374657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2117.389204] LNet: 122497:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2117.404511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2117.440689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2117.467837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2117.518877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2117.560848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2117.605209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2118.083636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2118.285275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2118.310455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2118.333493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2118.356626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2118.385608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2118.419820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2118.450696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2118.479562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2118.500943] LNet: 122614:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2118.521797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2118.556849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2118.587989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2118.623773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2118.648922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2118.687213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2119.202948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.206.122@tcp [ 2119.375054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2119.399476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2119.423403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2119.442721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2119.466659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2119.501262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2119.521707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2119.548978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2119.576530] LNet: 122731:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2119.596712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2119.624618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2119.654445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2119.679408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2119.717654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2119.758047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2120.309377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2120.511778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2120.541452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2120.572429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2120.617799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2120.652824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2120.679649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2120.699590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2120.727680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2120.753647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2120.777615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2120.804714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2120.828447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2120.863460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2120.894787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2121.229896] LNet: 122898:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2121.236559] LNetError: 122898:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2121.254103] LNet: Removed LNI 192.168.206.22@tcp [ 2121.561662] Key type .llcrypt unregistered [ 2121.567499] Key type ._llcrypt unregistered [ 2134.304538] Key type ._llcrypt registered [ 2134.306635] Key type .llcrypt registered [ 2135.378875] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 18:58:02 (1786748282) [ 2135.884679] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2135.898819] alg: No test for adler32 (adler32-zlib) [ 2136.847486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2137.213825] LNet: 123786:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2137.829367] Key type .llcrypt unregistered [ 2137.837246] Key type ._llcrypt unregistered [ 2142.945420] Key type ._llcrypt registered [ 2142.947213] Key type .llcrypt registered [ 2143.146442] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2143.162870] alg: No test for adler32 (adler32-zlib) [ 2144.043975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2144.067097] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 2150.151166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.122@tcp [ 2157.544374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2157.579437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2157.601382] LNet: Added LNI 192.168.206.22@tcp1 [8/256/0/180] [ 2158.402951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2158.851207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.206.122@tcp [ 2159.048661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2159.066498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2159.094463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2159.133565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2159.178407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2159.203551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2159.231796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2159.262276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2159.280321] LNet: 124442:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2159.294659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2159.332371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2159.365383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2159.412474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2159.456879] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.122@tcp [ 2159.477455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2159.523450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2160.080122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2160.290578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2160.336217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2160.380992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2160.415571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2160.461159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2160.501699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2160.535968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2160.574609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2160.595684] LNet: 124559:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2160.620234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2160.669805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2160.720499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2160.762553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2160.796964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2160.837177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2161.551398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.206.122@tcp [ 2161.855231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2161.902740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2161.938345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2161.994435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2162.079813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2162.124207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2162.176870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2162.221866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2162.249936] LNet: 124676:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2162.279291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2162.331424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2162.391211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2162.429588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2162.469973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2162.517927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2163.286578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2163.547610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2163.582588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2163.620876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2163.658066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2163.700627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2163.740827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2163.775355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2163.822954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2163.861518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2163.894206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2163.956645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2164.004752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2164.054751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2164.102251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2168.820207] LNet: 124879:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2168.827228] LNetError: 124879:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2168.837560] LNet: Removed LNI 192.168.206.22@tcp [ 2168.852966] LNet: Removed LNI 192.168.206.22@tcp1 [ 2169.190959] Key type .llcrypt unregistered [ 2169.193930] Key type ._llcrypt unregistered [ 2182.923812] Key type ._llcrypt registered [ 2182.930753] Key type .llcrypt registered [ 2183.959454] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 18:58:50 (1786748330) [ 2184.309714] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2184.324281] alg: No test for adler32 (adler32-zlib) [ 2185.236514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2185.299622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2185.322004] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 2186.038791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2186.076984] LNet: Added LNI 192.168.206.22@tcp2 [8/256/0/180] [ 2186.885705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.22@tcp [ 2186.971088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2188.000155] LNet: Removed LNI 192.168.206.22@tcp [ 2193.078513] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 18:59:00 (1786748340) [ 2194.338348] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2195.839842] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 18:59:02 (1786748342) [ 2202.253177] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 18:59:09 (1786748349) [ 2207.944357] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 18:59:14 (1786748354) [ 2213.832638] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 18:59:20 (1786748360) [ 2219.933657] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 18:59:27 (1786748367) [ 2225.928876] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 18:59:33 (1786748373) [ 2231.798865] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 18:59:38 (1786748378) [ 2237.427684] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 18:59:44 (1786748384) [ 2242.807822] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 18:59:49 (1786748389) [ 2248.605311] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 18:59:55 (1786748395) [ 2254.007634] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 19:00:00 (1786748400) [ 2260.997969] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 19:00:07 (1786748407) [ 2268.629635] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 19:00:15 (1786748415) [ 2276.292245] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 19:00:23 (1786748423) [ 2282.376405] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 19:00:29 (1786748429) [ 2289.182312] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 19:00:36 (1786748436) [ 2297.472558] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 19:00:44 (1786748444) [ 2305.176939] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 19:00:51 (1786748451) [ 2305.244742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2305.288863] LNet: 136561:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2305.302566] LNetError: 136561:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2305.327057] LNet: Removed LNI 192.168.206.22@tcp2 [ 2305.353023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2305.489316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2305.510728] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 2306.351938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2306.367344] LNet: 136712:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2306.371843] LNetError: 136712:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2306.399062] LNet: Removed LNI 192.168.206.22@tcp [ 2306.436039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2306.505702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2306.544226] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 2312.651305] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 19:00:59 (1786748459) [ 2320.782940] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 19:01:07 (1786748467) [ 2328.777863] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 19:01:15 (1786748475) [ 2336.194114] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 19:01:22 (1786748482) [ 2343.957967] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 19:01:30 (1786748490) [ 2351.227794] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 19:01:38 (1786748498) [ 2359.101756] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 19:01:45 (1786748505) [ 2365.533739] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 19:01:52 (1786748512) [ 2372.380878] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 19:01:59 (1786748519) [ 2379.628871] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 19:02:06 (1786748526) [ 2387.237683] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 19:02:13 (1786748533) [ 2394.482290] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 19:02:21 (1786748541) [ 2403.126476] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2187 sec ========== 19:02:29 (1786748549) [ 2403.444836] LNet: 144193:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2403.454671] LNetError: 144193:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2403.487506] LNet: Removed LNI 192.168.206.22@tcp [ 2404.560150] Key type .llcrypt unregistered [ 2404.568630] Key type ._llcrypt unregistered [ 2417.419237] Key type ._llcrypt registered [ 2417.424878] Key type .llcrypt registered [ 2418.041468] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2418.057769] alg: No test for adler32 (adler32-zlib) [ 2419.200888] Lustre: Lustre: Build Version: 2.17.57_3_g6085841 [ 2419.409989] LNet: Added LNI 192.168.206.22@tcp [8/256/0/180] [ 2421.096218] Key type lgssc registered [ 2422.761841] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2486.101163] Lustre: Mounted lustre-client - version 2.17.57_3_g6085841 [ 2491.523653] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2499.494770] Lustre: Unmounted lustre-client [ 2525.335258] Key type lgssc unregistered [ 2525.577792] LNet: 147152:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2525.589436] LNetError: 147152:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2525.604694] LNet: Removed LNI 192.168.206.22@tcp [ 2526.178312] Key type .llcrypt unregistered [ 2526.180624] Key type ._llcrypt unregistered