[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 449493476 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524588K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002324] x2apic enabled [ 0.003006] Switched APIC routing to physical x2apic. [ 0.004010] kvm-guest: setup PV IPIs [ 0.007484] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008018] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010006] pid_max: default: 32768 minimum: 301 [ 0.011130] LSM: Security Framework initializing [ 0.012041] Yama: becoming mindful. [ 0.013027] SELinux: Initializing. [ 0.014054] *** VALIDATE selinux *** [ 0.021721] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025509] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026127] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027086] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028093] *** VALIDATE tmpfs *** [ 0.030299] *** VALIDATE proc *** [ 0.031194] *** VALIDATE cgroup *** [ 0.032007] *** VALIDATE cgroup2 *** [ 0.033241] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035131] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037023] Spectre V2 : User space: Vulnerable [ 0.038007] Speculative Store Bypass: Vulnerable [ 0.041153] debug: unmapping init [mem 0xffffffffad659000-0xffffffffad660fff] [ 0.043390] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045140] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046021] ... version: 2 [ 0.046997] ... bit width: 48 [ 0.047011] ... generic registers: 4 [ 0.047944] ... value mask: 0000ffffffffffff [ 0.048013] ... max period: 00007fffffffffff [ 0.049010] ... fixed-purpose events: 3 [ 0.050010] ... event mask: 000000070000000f [ 0.051249] rcu: Hierarchical SRCU implementation. [ 0.053316] smp: Bringing up secondary CPUs ... [ 0.054509] x86: Booting SMP configuration: [ 0.055022] .... node #0, CPUs: #1 #2 #3 [ 0.060558] smp: Brought up 1 node, 4 CPUs [ 0.062010] smpboot: Max logical packages: 1 [ 0.063013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.140514] node 0 deferred pages initialised in 74ms [ 0.143085] devtmpfs: initialized [ 0.144300] x86/mm: Memory block size: 128MB [ 0.147914] gcov: version magic: 0x41383552 [ 0.151105] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.152276] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.153303] pinctrl core: initialized pinctrl subsystem [ 0.154229] [ 0.154876] ************************************************************* [ 0.155011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.156012] ** ** [ 0.157009] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.158012] ** ** [ 0.159012] ** This means that this kernel is built to expose internal ** [ 0.160012] ** IOMMU data structures, which may compromise security on ** [ 0.161011] ** your system. ** [ 0.162010] ** ** [ 0.163013] ** If you see this message and you are not debugging the ** [ 0.164011] ** kernel, report this immediately to your vendor! ** [ 0.165014] ** ** [ 0.166011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.167014] ************************************************************* [ 0.168900] NET: Registered protocol family 16 [ 0.170421] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.173058] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.176059] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.181073] cpuidle: using governor menu [ 0.183914] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.185424] PCI: Using configuration type 1 for base access [ 0.188126] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.198054] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.199022] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.200152] cryptd: max_cpu_qlen set to 1000 [ 0.203411] ACPI: Added _OSI(Module Device) [ 0.205013] ACPI: Added _OSI(Processor Device) [ 0.206012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.208011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.214626] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.220241] ACPI: Interpreter enabled [ 0.221049] ACPI: PM: (supports S0 S3 S4 S5) [ 0.223010] ACPI: Using IOAPIC for interrupt routing [ 0.224082] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.227385] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.239679] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.242038] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.245016] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.248171] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.254219] acpiphp: Slot [2] registered [ 0.256222] acpiphp: Slot [5] registered [ 0.257102] acpiphp: Slot [6] registered [ 0.259095] acpiphp: Slot [3] registered [ 0.260080] acpiphp: Slot [4] registered [ 0.261084] acpiphp: Slot [7] registered [ 0.263121] acpiphp: Slot [8] registered [ 0.264082] acpiphp: Slot [9] registered [ 0.265098] acpiphp: Slot [10] registered [ 0.267088] acpiphp: Slot [11] registered [ 0.268085] acpiphp: Slot [12] registered [ 0.269079] acpiphp: Slot [13] registered [ 0.271085] acpiphp: Slot [14] registered [ 0.272155] acpiphp: Slot [15] registered [ 0.273163] acpiphp: Slot [16] registered [ 0.275084] acpiphp: Slot [17] registered [ 0.276084] acpiphp: Slot [18] registered [ 0.277078] acpiphp: Slot [19] registered [ 0.279088] acpiphp: Slot [20] registered [ 0.280079] acpiphp: Slot [21] registered [ 0.281103] acpiphp: Slot [22] registered [ 0.283085] acpiphp: Slot [23] registered [ 0.284076] acpiphp: Slot [24] registered [ 0.285074] acpiphp: Slot [25] registered [ 0.287074] acpiphp: Slot [26] registered [ 0.288191] acpiphp: Slot [27] registered [ 0.289084] acpiphp: Slot [28] registered [ 0.291193] acpiphp: Slot [29] registered [ 0.292073] acpiphp: Slot [30] registered [ 0.293086] acpiphp: Slot [31] registered [ 0.295072] PCI host bridge to bus 0000:00 [ 0.296015] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.298075] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.300022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.303019] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.305019] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.307020] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.309181] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.312078] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.315423] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.323014] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.327054] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.330021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.332014] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.334015] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.336559] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.339896] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.342046] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.344855] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.349014] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.360016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.364679] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.368492] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.373949] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.382009] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.393015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.409695] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.418023] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.423016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.438020] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.450147] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.451446] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.454427] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.459485] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.463425] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.470767] iommu: Default domain type: Passthrough [ 0.472545] SCSI subsystem initialized [ 0.473223] ACPI: bus type USB registered [ 0.475119] usbcore: registered new interface driver usbfs [ 0.476116] usbcore: registered new interface driver hub [ 0.478129] usbcore: registered new device driver usb [ 0.480200] pps_core: LinuxPPS API ver. 1 registered [ 0.481011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.484092] PTP clock support registered [ 0.486096] EDAC MC: Ver: 3.0.0 [ 0.488153] PCI: Using ACPI for IRQ routing [ 0.489000] NetLabel: Initializing [ 0.490016] NetLabel: domain hash size = 128 [ 0.492011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.494080] NetLabel: unlabeled traffic allowed by default [ 0.497004] vgaarb: loaded [ 0.498875] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.501010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.507362] clocksource: Switched to clocksource kvm-clock [ 0.622191] VFS: Disk quotas dquot_6.6.0 [ 0.623570] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.625569] *** VALIDATE ramfs *** [ 0.626382] *** VALIDATE hugetlbfs *** [ 0.627615] pnp: PnP ACPI init [ 0.630694] pnp: PnP ACPI: found 6 devices [ 0.650149] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.653128] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.654931] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.656722] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.658766] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.660844] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.663450] NET: Registered protocol family 2 [ 0.666042] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.671489] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.674830] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.679549] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.682452] TCP: Hash tables configured (established 65536 bind 65536) [ 0.685269] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.688923] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.691959] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.695221] NET: Registered protocol family 1 [ 0.701075] RPC: Registered named UNIX socket transport module. [ 0.703318] RPC: Registered udp transport module. [ 0.705105] RPC: Registered tcp transport module. [ 0.709964] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.714859] NET: Registered protocol family 44 [ 0.716197] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.719221] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.724306] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.726158] PCI: CLS 0 bytes, default 64 [ 0.728194] Unpacking initramfs... [ 2.216808] debug: unmapping init [mem 0xffff94257cc64000-0xffff94257ffcffff] [ 2.223242] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.225175] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.229088] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.757133] Initialise system trusted keyrings [ 2.759229] Key type blacklist registered [ 2.762522] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.772037] zbud: loaded [ 2.775404] *** VALIDATE nfs *** [ 2.776778] *** VALIDATE nfs4 *** [ 2.778922] pstore: using deflate compression [ 2.783228] Platform Keyring initialized [ 2.908585] NET: Registered protocol family 38 [ 2.910654] Key type asymmetric registered [ 2.912599] Asymmetric key parser 'x509' registered [ 2.917620] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.920663] io scheduler mq-deadline registered [ 2.922628] io scheduler kyber registered [ 2.924370] io scheduler bfq registered [ 2.926393] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.929905] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.936458] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.939473] ACPI: Power Button [PWRF] [ 2.945452] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.952811] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.964817] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.994182] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.025931] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.044466] Non-volatile memory driver v1.3 [ 3.046214] Linux agpgart interface v0.103 [ 3.090746] virtio_blk virtio1: [vda] 146672 512-byte logical blocks (75.1 MB/71.6 MiB) [ 3.094348] vda: detected capacity change from 0 to 75096064 [ 3.110342] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.112744] vdb: detected capacity change from 0 to 1073741824 [ 3.118827] libphy: Fixed MDIO Bus: probed [ 3.124261] usbcore: registered new interface driver usbserial_generic [ 3.126100] usbserial: USB Serial support registered for generic [ 3.128366] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.132022] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.133966] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.136363] mousedev: PS/2 mouse device common for all mice [ 3.138633] rtc_cmos 00:05: RTC can wake from S4 [ 3.142707] rtc_cmos 00:05: registered as rtc0 [ 3.144644] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.147621] intel_pstate: CPU model not supported [ 3.149750] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.154309] hid: raw HID events driver (C) Jiri Kosina [ 3.156589] usbcore: registered new interface driver usbhid [ 3.157389] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.158907] usbhid: USB HID core driver [ 3.163232] drop_monitor: Initializing network drop monitor service [ 3.165893] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.166200] Initializing XFRM netlink socket [ 3.171489] NET: Registered protocol family 10 [ 3.174261] Segment Routing with IPv6 [ 3.175708] NET: Registered protocol family 17 [ 3.177764] mpls_gso: MPLS GSO support [ 3.185220] RAS: Correctable Errors collector initialized. [ 3.187697] AVX version of gcm_enc/dec engaged. [ 3.189629] AES CTR mode by8 optimization enabled [ 3.286758] sched_clock: Marking stable (3286737435, 0)->(4152818655, -866081220) [ 3.290429] registered taskstats version 1 [ 3.292631] Loading compiled-in X.509 certificates [ 3.295651] zswap: loaded using pool lzo/zbud [ 3.321186] Key type big_key registered [ 3.334439] Key type encrypted registered [ 3.336295] ima: No TPM chip found, activating TPM-bypass! [ 3.338536] ima: Allocated hash algorithm: sha1 [ 3.340383] ima: No architecture policies found [ 3.342206] evm: Initialising EVM extended attributes: [ 3.344189] evm: security.selinux [ 3.345543] evm: security.ima [ 3.346759] evm: security.capability [ 3.348304] evm: HMAC attrs: 0x1 [ 3.350634] rtc_cmos 00:05: setting system clock to 2026-09-08 02:51:45 UTC (1788835905) [ 3.357470] debug: unmapping init [mem 0xffffffffae603000-0xffffffffae7fffff] [ 3.360974] debug: unmapping init [mem 0xffffffffad382000-0xffffffffad658fff] [ 3.370113] Write protecting the kernel read-only data: 28672k [ 3.373476] debug: unmapping init [mem 0xffffffffaba03000-0xffffffffabbfffff] [ 3.376631] debug: unmapping init [mem 0xffffffffac314000-0xffffffffac3fffff] [ 3.409519] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.418822] systemd[1]: Detected virtualization kvm. [ 3.420849] systemd[1]: Detected architecture x86-64. [ 3.422930] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.448390] systemd[1]: No hostname configured. [ 3.450255] systemd[1]: Set hostname to . [ 3.452273] random: systemd: uninitialized urandom read (16 bytes read) [ 3.454395] systemd[1]: Initializing machine ID from random generator. [ 3.521069] random: ln: uninitialized urandom read (6 bytes read) [ 3.593909] random: systemd: uninitialized urandom read (16 bytes read) [ 3.596525] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.602892] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.611832] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Initrd Root Device. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Swap. [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Timers. Starting Apply Kernel Variables... [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.281711] device-mapper: uevent: version 1.0.3 [ 4.285173] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.126032] virtio_net virtio0 ens2: renamed from eth0 [ 5.178054] scsi host0: ata_piix [ 5.188222] scsi host1: ata_piix [ 5.191178] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.194471] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.672355] dracut-initqueue[587]: RTNETLINK answers: File exists [ 9.853179] random: crng init done [ 9.854684] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.449917] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.719246] printk: systemd: 22 output lines suppressed due to ratelimiting [ 11.997705] SELinux: Disabled at runtime. [ 12.059675] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.069115] systemd[1]: Detected virtualization kvm. [ 12.071134] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.651187] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.654509] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.658412] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.661711] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.665163] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.673647] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.684610] systemd[1]: Mounting Huge Pages File System... Mounting Huge Pages File System... [ OK ] Listening on udev Control Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-getty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Created slice User and Session Slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Initrd Root File System. Activating swap /dev/disk/by-label/SWAP... Mounting Kernel Debug File System... [ OK ] Created slice system-serial\x2dgetty.slice. Mounting POSIX Message Queue File System... [ 12.789028] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Apply Kernel Variables... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target Slices. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 13.101838] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 13.515574] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.557297] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.706658] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.721537] EDAC sbridge: Ver: 1.1.2 [ 14.912994] Key type dns_resolver registered [ 15.233508] NFS: Registering the id_resolver key type [ 15.236066] Key type id_resolver registered [ 15.237684] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started Login Service. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg632-client login: [ 45.923221] libcfs: loading out-of-tree module taints kernel. [ 46.114403] Key type ._llcrypt registered [ 46.152217] Key type .llcrypt registered [ 46.767639] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 46.786759] alg: No test for adler32 (adler32-zlib) [ 48.607927] Lustre: Lustre: Build Version: 2.17.57_107_gb4eb68b [ 49.544577] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 51.359243] Key type lgssc registered [ 53.858931] Lustre: Echo OBD driver; http://www.lustre.org/ [ 177.338523] Lustre: Mounted lustre-client - version 2.17.57_107_gb4eb68b [ 181.918294] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 200.299910] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing check_logdir /tmp/testlogs/ [ 201.200970] Lustre: lustre-OST0000-osc-ffff9425e06ef800: disconnect after 22s idle [ 204.045612] hrtimer: interrupt took 13298340 ns [ 205.175994] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing yml_node [ 209.864442] Lustre: DEBUG MARKER: Client: 2.17.57.107 [ 212.302529] Lustre: DEBUG MARKER: MDS: 2.17.57.107 [ 215.228561] Lustre: DEBUG MARKER: OSS: 2.17.57.107 [ 216.617649] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Mon Sep 7 22:55:17 EDT 2026 [ 231.784314] Lustre: Unmounted lustre-client [ 258.753627] Key type lgssc unregistered [ 259.051791] LNet: 8440:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 259.064377] LNetError: 8440:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 259.089484] LNet: Removed LNI 192.168.206.32@tcp [ 259.683401] Key type .llcrypt unregistered [ 259.685758] Key type ._llcrypt unregistered [ 284.815861] Key type ._llcrypt registered [ 284.818702] Key type .llcrypt registered [ 284.998189] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 285.014958] alg: No test for adler32 (adler32-zlib) [ 285.986267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 286.072617] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 286.107421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 289.788536] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 289.886928] LNet: 9310:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 289.898748] LNetError: 9310:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 289.915780] LNet: Removed LNI 192.168.206.32@tcp [ 290.455069] Key type .llcrypt unregistered [ 290.457842] Key type ._llcrypt unregistered [ 296.222692] Key type ._llcrypt registered [ 296.223845] Key type .llcrypt registered [ 297.436619] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 22:56:38 (1788836198) [ 297.901761] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 297.913215] alg: No test for adler32 (adler32-zlib) [ 298.829160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 298.897760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 298.912832] LNet: 9729:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 298.927606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 298.994383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 299.016478] LNet: 9736:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 305.754644] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 22:56:46 (1788836206) [ 306.052208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 306.075309] LNet: 10313:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 306.109335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 306.226251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 312.765661] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 22:56:53 (1788836213) [ 312.902842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 312.937741] LNet: 10895:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 312.974626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 313.046119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 319.409286] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 22:57:00 (1788836220) [ 319.487057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 319.526902] LNet: 11475:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 319.560939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 319.651644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 326.408688] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 22:57:07 (1788836227) [ 326.721498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 326.743723] LNet: 12067:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 326.765212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 326.861353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 327.263649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 333.434878] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 22:57:14 (1788836234) [ 333.628066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 333.718099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 333.795452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 334.347190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 334.809357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 340.615713] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 22:57:21 (1788836241) [ 340.669397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 340.700961] LNet: 13249:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 340.722851] LNet: 13249:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 340.750501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 340.823307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 346.778137] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 22:57:27 (1788836247) [ 346.970894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 347.001299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 347.061530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 347.117856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 347.530832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 347.592907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 348.126927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 348.173578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 348.617480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 348.642941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 349.103399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 349.149468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 349.711114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 349.743617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 350.331575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 350.404987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 350.960439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 351.017879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 351.561653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 351.633979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 352.146704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 352.180525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 352.662772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 352.709984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 353.231964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 353.288710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 353.877581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 353.958181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 359.882618] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 22:57:40 (1788836260) [ 360.042479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 360.062524] LNet: 14523:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 360.072659] LNet: 14523:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 360.108757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 360.232565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 360.267049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 366.118690] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 22:57:47 (1788836267) [ 366.263950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 366.313992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 366.360143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 366.389050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 371.665475] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 22:57:52 (1788836272) [ 371.967359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 372.006693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 372.078475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 372.100843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 380.173980] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 22:58:00 (1788836280) [ 380.358908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 380.443777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 380.577221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 380.644601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 388.487336] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 22:58:08 (1788836288) [ 388.876864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 388.950805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 389.055463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 389.083683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 396.067930] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 22:58:17 (1788836297) [ 396.372328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 396.397434] LNet: 17457:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 396.401503] LNet: 17457:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 396.432513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 396.515886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 396.563974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 403.404403] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 22:58:24 (1788836304) [ 403.629648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 403.665681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 403.783618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 404.311676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 412.094206] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 22:58:32 (1788836312) [ 412.257284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 412.324332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 412.600890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 412.839226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 419.275696] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 22:58:40 (1788836320) [ 419.487878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 419.524549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 419.736836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 419.991625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 426.801877] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 22:58:47 (1788836327) [ 426.923550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 426.994127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 427.496996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 427.791216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 434.162292] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 22:58:54 (1788836334) [ 434.314115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 434.353594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 434.443162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 434.512897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 440.787641] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 22:59:01 (1788836341) [ 440.931388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 440.998215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 441.062756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 441.103363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 447.472262] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 22:59:08 (1788836348) [ 447.596085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 447.655964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 447.755739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 454.168563] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 22:59:14 (1788836354) [ 454.265429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 454.342496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 454.438626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 460.943638] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 22:59:21 (1788836361) [ 461.141373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 461.163937] LNet: 22894:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 461.171877] LNet: 22894:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 461.196278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 461.271765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 461.749392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 468.210138] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 22:59:29 (1788836369) [ 468.309616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 468.359134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 468.496382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 468.542674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 474.624621] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 22:59:35 (1788836375) [ 474.765363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 474.813704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 474.888279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 474.920965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 480.209856] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 22:59:41 (1788836381) [ 480.414955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 480.451486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 480.514358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 480.548794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 486.384607] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 22:59:47 (1788836387) [ 486.564671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 486.634663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 486.702677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 486.728603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 492.849302] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 22:59:53 (1788836393) [ 493.016493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 493.063828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 493.119768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 493.173147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 493.196837] LNetError: 25843:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 493.234339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 493.280058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 499.255266] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 23:00:00 (1788836400) [ 499.402084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 499.487384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 499.544984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.DxfoyDGcVF [ 499.617286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.DxfoyDGcVF [ 499.664323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.DxfoyDGcVF [ 505.107411] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 23:00:06 (1788836406) [ 505.251955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 505.295935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 505.352968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 505.401548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 505.452749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 511.026668] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 23:00:11 (1788836411) [ 511.170371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 511.236280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 511.286852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 511.331754] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 512.747600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.32@tcp [ 513.934867] Lustre: Lustre: Build Version: 2.17.57_107_gb4eb68b [ 515.863179] Key type lgssc registered [ 516.843881] Lustre: Echo OBD driver; http://www.lustre.org/ [ 520.150384] LNetError: 27604:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.206.32@tcp99 has no available nets [ 522.561638] Key type lgssc unregistered [ 522.906141] LNetError: 28709:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 522.921750] LNet: Removed LNI 192.168.206.32@tcp [ 523.452734] Key type .llcrypt unregistered [ 523.457365] Key type ._llcrypt unregistered [ 529.329847] Key type ._llcrypt registered [ 529.333386] Key type .llcrypt registered [ 530.555910] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 23:00:31 (1788836431) [ 530.907973] Key type .llcrypt unregistered [ 530.914683] Key type ._llcrypt unregistered [ 536.265685] Key type ._llcrypt registered [ 536.268306] Key type .llcrypt registered [ 536.404849] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 536.424760] alg: No test for adler32 (adler32-zlib) [ 537.277867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 537.339383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 537.361578] LNet: 29624:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 537.370246] LNet: 29624:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 537.379886] LNet: 29624:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 537.397322] LNet: Message forwarding will be enabled in 2s [ 540.511162] LNet: Message forwarding enabled [ 540.852745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 540.877101] LNet: Message forwarding will stop in 2s to 6s [ 544.611198] LNet: Message forwarding disabled [ 545.312896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 545.357753] LNet: 29722:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 545.366726] LNet: 29722:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 545.377898] LNet: 29722:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 545.387941] LNet: Message forwarding will be enabled in 2s [ 545.515791] LNet: 29735:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 546.149629] Key type .llcrypt unregistered [ 546.154661] Key type ._llcrypt unregistered [ 557.045774] Key type ._llcrypt registered [ 557.050952] Key type .llcrypt registered [ 558.487735] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 23:00:58 (1788836458) [ 559.014253] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 559.082162] alg: No test for adler32 (adler32-zlib) [ 560.141807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 560.284895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 566.707978] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 23:01:07 (1788836467) [ 567.003054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 567.029593] LNet: 31147:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 567.061515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 567.138724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 567.219603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 567.265383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 567.315871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 567.375616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 567.459433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 567.511225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 567.571343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 567.649326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 567.703905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 567.743525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 567.792778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 567.851858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 567.905254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 567.936497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 567.977720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 568.034410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 568.103351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 568.152152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 568.191794] LNetError: 31190:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 573.944501] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 23:01:14 (1788836474) [ 574.064612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 574.093854] LNet: 31756:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 574.125312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 574.248285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 580.674832] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 23:01:21 (1788836481) [ 582.286467] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg632-server [ 583.880181] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 23:01:24 (1788836484) [ 584.057526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 584.104557] LNet: 32689:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 584.155399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 584.216687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 584.250230] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 585.276519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.206.[33-37]@tcp [ 585.292396] LNet: 32825:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 585.334431] LNetError: 32695:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.33@tcp is being used as a gateway but routing feature is not turned on [ 585.363204] LNetError: 32695:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp1 through 192.168.206.33@tcp has gone from up to down [ 590.089981] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 23:01:31 (1788836491) [ 591.691450] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg632-server [ 593.041759] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 23:01:34 (1788836494) [ 593.113900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 593.140951] LNet: 33756:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 593.153385] LNetError: 33756:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 594.222515] LNet: Removed LNI 192.168.206.32@tcp [ 594.255418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 594.328800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 594.362946] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 594.986808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.206.[33-38/2]@tcp [ 595.008666] LNet: 33877:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 595.017847] LNet: 33877:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 595.034201] LNetError: 33764:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.33@tcp is being used as a gateway but routing feature is not turned on [ 595.045532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.206.[33-38/2]@tcp [ 595.046851] LNetError: 33764:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 595.046870] LNetError: 33764:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp103 through 192.168.206.33@tcp has gone from up to down [ 595.046874] LNetError: 33764:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 601.540619] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 23:01:42 (1788836502) [ 601.610909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 601.632960] LNet: 34464:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 601.639098] LNetError: 34464:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 601.673650] LNet: Removed LNI 192.168.206.32@tcp [ 601.699635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 601.872678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 601.934491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 601.965710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 602.017633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 602.073157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 602.108328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 602.165937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 602.254090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 602.300616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 602.378602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 602.488974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 602.532206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 602.581977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 602.696087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 602.791996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 602.862705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 602.935187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 602.968597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 603.013502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 603.098318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 603.146068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 603.200627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 603.279478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 603.315486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 603.347570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 603.411800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 603.432404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 603.469267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 603.546203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 603.585959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 603.630648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 603.687387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 603.714431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 603.799460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 609.845313] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 23:01:50 (1788836510) [ 611.518510] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg632-server [ 613.229707] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 23:01:54 (1788836514) [ 614.787632] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg632-server [ 616.881146] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 23:01:57 (1788836517) [ 617.119640] LNet: 35870:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 617.124804] LNet: 35870:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 617.661438] Key type .llcrypt unregistered [ 617.666383] Key type ._llcrypt unregistered [ 622.723229] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 622.808616] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 622.818188] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 622.827656] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 625.362880] Key type ._llcrypt registered [ 625.365413] Key type .llcrypt registered [ 625.519490] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 625.528177] alg: No test for adler32 (adler32-zlib) [ 626.471823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 626.524338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 626.556811] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 627.015687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 627.028368] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 628.620457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 628.636389] LNet: Removed LNI 10.1.2.3@tcp [ 628.740896] LNet: 36588:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 628.745485] LNetError: 36588:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 628.756027] LNet: Removed LNI 192.168.206.32@tcp [ 629.188630] Key type .llcrypt unregistered [ 629.190608] Key type ._llcrypt unregistered [ 637.469829] Key type ._llcrypt registered [ 637.472342] Key type .llcrypt registered [ 638.695427] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 23:02:19 (1788836539) [ 639.092486] Key type .llcrypt unregistered [ 639.099114] Key type ._llcrypt unregistered [ 644.112950] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 644.246177] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 644.259801] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 644.263336] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 646.697912] Key type ._llcrypt registered [ 646.703300] Key type .llcrypt registered [ 646.846359] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 646.857553] alg: No test for adler32 (adler32-zlib) [ 647.793672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 647.845511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 647.872324] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 648.496687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 648.519378] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 650.647950] LNet: 38071:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 650.656734] LNetError: 38071:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 650.666747] LNet: Removed LNI 10.1.2.3@tcp [ 650.686092] LNet: Removed LNI 192.168.206.32@tcp [ 651.278359] Key type .llcrypt unregistered [ 651.281403] Key type ._llcrypt unregistered [ 661.223898] Key type ._llcrypt registered [ 661.226228] Key type .llcrypt registered [ 662.391509] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 23:02:43 (1788836563) [ 662.783360] Key type .llcrypt unregistered [ 662.786564] Key type ._llcrypt unregistered [ 668.429891] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 668.545034] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 668.554094] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 668.559208] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 670.151567] Key type ._llcrypt registered [ 670.153256] Key type .llcrypt registered [ 670.341233] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 670.352557] alg: No test for adler32 (adler32-zlib) [ 671.394739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 671.458718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 671.483702] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 673.361442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 673.396553] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 675.575760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 675.594965] LNet: Removed LNI 10.1.2.3@tcp [ 675.611764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 675.644662] LNet: Removed LNI 10.1.2.31@tcp [ 675.659874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 675.694276] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 676.302920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 676.327368] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 677.124197] LNet: 39979:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 677.133956] LNetError: 39979:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 677.140205] LNet: Removed LNI 10.1.2.31@tcp [ 677.156116] LNet: Removed LNI 10.1.2.3@tcp [ 677.661307] Key type .llcrypt unregistered [ 677.663087] Key type ._llcrypt unregistered [ 686.275417] Key type ._llcrypt registered [ 686.278135] Key type .llcrypt registered [ 687.248437] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 23:03:08 (1788836588) [ 687.594124] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 687.607462] alg: No test for adler32 (adler32-zlib) [ 688.501887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 688.557588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 688.587438] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 689.114756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 689.127974] LNet: 40928:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 689.132334] LNetError: 40928:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 689.147153] LNet: Removed LNI 192.168.206.32@tcp [ 689.163204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 689.212182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.206.32@tcp [ 689.235928] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 689.758575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.206.32@tcp [ 689.791966] LNet: Removed LNI 192.168.206.32@tcp [ 689.858551] LNet: 41063:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 690.326556] Key type .llcrypt unregistered [ 690.329103] Key type ._llcrypt unregistered [ 699.627662] Key type ._llcrypt registered [ 699.630808] Key type .llcrypt registered [ 700.693313] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 23:03:21 (1788836601) [ 702.245144] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg632-server [ 703.789340] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 23:03:24 (1788836604) [ 704.173506] Key type .llcrypt unregistered [ 704.176699] Key type ._llcrypt unregistered [ 709.239565] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 709.406918] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 709.431386] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 709.438328] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 711.843889] Key type ._llcrypt registered [ 711.846991] Key type .llcrypt registered [ 712.084094] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 712.097577] alg: No test for adler32 (adler32-zlib) [ 713.021570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 713.099142] LNet: Added LNI 192.168.206.32@tcp1 [8/256/0/180] [ 713.694397] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 715.616319] LNet: 42894:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 715.628552] LNetError: 42894:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 715.633542] LNet: Removed LNI 192.168.206.32@tcp1 [ 715.643889] LNet: Removed LNI 10.1.2.3@tcp2 [ 716.180404] Key type .llcrypt unregistered [ 716.182160] Key type ._llcrypt unregistered [ 725.897393] Key type ._llcrypt registered [ 725.900666] Key type .llcrypt registered [ 727.048988] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 23:03:47 (1788836627) [ 727.322266] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 727.337535] alg: No test for adler32 (adler32-zlib) [ 728.246706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 728.298839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 728.317668] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 728.996703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 729.011765] LNet: 43860:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 729.020759] LNetError: 43860:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 729.040428] LNet: Removed LNI 192.168.206.32@tcp [ 729.059676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 729.117957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 729.137676] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 734.477816] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 23:03:55 (1788836635) [ 734.581564] LNet: 44444:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 734.591255] LNetError: 44444:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 734.604751] LNet: Removed LNI 192.168.206.32@tcp [ 735.160768] Key type .llcrypt unregistered [ 735.165167] Key type ._llcrypt unregistered [ 740.495227] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 740.618372] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 740.633802] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 740.642918] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 742.060387] Key type ._llcrypt registered [ 742.063940] Key type .llcrypt registered [ 742.262857] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 742.281493] alg: No test for adler32 (adler32-zlib) [ 743.217819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 743.280063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 743.327795] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 743.961574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 743.992562] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 746.079600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 746.096801] LNet: 45175:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 746.101333] LNetError: 45175:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 746.106673] LNet: Removed LNI 10.1.2.3@tcp [ 746.119693] LNet: Removed LNI 192.168.206.32@tcp [ 746.138596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 746.186636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 746.214634] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 746.220369] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 746.753931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 746.786719] LNet: 45202:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 746.796309] LNetError: 45202:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 746.805233] LNetError: 45202:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 746.809996] LNet: Removed LNI 10.1.2.3@tcp [ 746.823322] LNet: Removed LNI 192.168.206.32@tcp [ 746.872027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 746.956715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 746.988442] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 746.998309] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 747.398750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 747.413017] LNet: Added LNI 192.168.206.32@tcp2 [8/256/0/180] [ 748.281792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 748.308625] LNet: 45365:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 748.316764] LNetError: 45365:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 748.323525] LNetError: 45365:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 748.332623] LNet: Removed LNI 10.1.2.3@tcp [ 748.337253] LNet: Removed LNI 192.168.206.32@tcp [ 748.349175] LNet: Removed LNI 192.168.206.32@tcp2 [ 748.369173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 748.455475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 748.511559] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 748.515112] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 748.529160] LNet: Added LNI 192.168.206.32@tcp2 [8/256/0/180] [ 754.182931] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 23:04:15 (1788836655) [ 754.278222] LNet: 45955:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 754.289715] LNetError: 45955:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 754.299234] LNetError: 45955:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 754.305329] LNet: Removed LNI 10.1.2.3@tcp [ 754.308600] LNet: Removed LNI 192.168.206.32@tcp [ 754.346480] LNet: Removed LNI 192.168.206.32@tcp2 [ 754.890687] Key type .llcrypt unregistered [ 754.896833] Key type ._llcrypt unregistered [ 760.543818] Key type ._llcrypt registered [ 760.547801] Key type .llcrypt registered [ 760.755970] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 760.777072] alg: No test for adler32 (adler32-zlib) [ 761.793619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 761.853563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 761.893505] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 762.705957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 762.732406] LNet: 46355:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 762.738564] LNetError: 46355:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 762.751185] LNet: Removed LNI 192.168.206.32@tcp [ 762.787703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 762.852763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 762.878928] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 763.333348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 763.351747] LNet: 46385:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 763.359791] LNetError: 46385:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 763.375142] LNet: Removed LNI 192.168.206.32@tcp [ 763.393723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 763.459283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 763.485531] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 763.975885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 763.998738] LNet: Removed LNI 192.168.206.32@tcp [ 764.013794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 764.059234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 764.077385] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 764.514743] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 764.671831] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 764.678173] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 764.687218] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 767.018791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 767.033164] LNet: 46493:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 767.042103] LNet: 46493:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 767.050280] LNetError: 46493:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 767.056892] LNetError: 46493:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 767.069557] LNet: Removed LNI 192.168.206.32@tcp [ 767.096364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 767.172853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 767.195993] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 773.551206] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 23:04:34 (1788836674) [ 773.717950] LNet: 47086:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 773.725568] LNetError: 47086:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 773.752849] LNet: Removed LNI 192.168.206.32@tcp [ 774.487864] Key type .llcrypt unregistered [ 774.489420] Key type ._llcrypt unregistered [ 779.490826] Key type ._llcrypt registered [ 779.492554] Key type .llcrypt registered [ 779.621983] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 779.632615] alg: No test for adler32 (adler32-zlib) [ 780.548716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 780.597480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 780.616500] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 781.182236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 781.197358] LNet: 47485:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 781.207246] LNetError: 47485:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 781.225121] LNet: Removed LNI 192.168.206.32@tcp [ 781.244428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 781.303891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 781.327559] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 786.271061] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 23:04:47 (1788836687) [ 786.422465] LNet: 48073:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 786.429178] LNetError: 48073:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 786.440939] LNet: Removed LNI 192.168.206.32@tcp [ 787.000611] Key type .llcrypt unregistered [ 787.003469] Key type ._llcrypt unregistered [ 792.618217] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 792.736806] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 792.743795] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 792.749610] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 795.300913] Key type ._llcrypt registered [ 795.303375] Key type .llcrypt registered [ 795.435509] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 795.448115] alg: No test for adler32 (adler32-zlib) [ 796.384806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 796.436891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 796.456539] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 797.159653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 797.190991] LNet: 48523:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 797.202364] LNetError: 48523:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 797.225688] LNet: Removed LNI 192.168.206.32@tcp [ 797.243869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 797.279789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 797.302458] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 802.567646] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 23:05:03 (1788836703) [ 802.685314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 802.724520] LNet: 49116:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 802.744692] LNetError: 49116:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 802.780536] LNet: Removed LNI 192.168.206.32@tcp [ 802.809095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 802.943957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 808.352799] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 23:05:09 (1788836709) [ 808.578761] LNet: 49704:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 809.188751] Key type .llcrypt unregistered [ 809.192055] Key type ._llcrypt unregistered [ 814.605590] Key type ._llcrypt registered [ 814.609790] Key type .llcrypt registered [ 814.772098] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 814.784059] alg: No test for adler32 (adler32-zlib) [ 815.666545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 815.716432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 815.741913] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 816.446769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 816.474397] LNet: 50098:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 816.481105] LNetError: 50098:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 816.495893] LNet: Removed LNI 192.168.206.32@tcp [ 816.516883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 816.596382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 816.628233] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 821.849329] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 23:05:22 (1788836722) [ 822.034597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 822.059573] LNet: 50691:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 822.065650] LNetError: 50691:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 822.081407] LNet: Removed LNI 192.168.206.32@tcp [ 822.098708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 822.163869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 827.314687] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 23:05:28 (1788836728) [ 828.498428] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 829.981438] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 23:05:31 (1788836731) [ 831.210944] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 832.512625] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 23:05:33 (1788836733) [ 833.677841] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 835.098650] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 23:05:36 (1788836736) [ 836.327896] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 837.514861] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 23:05:38 (1788836738) [ 838.718414] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 840.041727] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 23:05:41 (1788836741) [ 840.184411] LNet: 52739:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 840.765193] Key type .llcrypt unregistered [ 840.766940] Key type ._llcrypt unregistered [ 845.776615] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 845.901263] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 845.923354] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 845.932873] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 848.483415] Key type ._llcrypt registered [ 848.485258] Key type .llcrypt registered [ 848.619234] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 848.633466] alg: No test for adler32 (adler32-zlib) [ 849.537856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 849.598285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 849.613519] LNet: Added LNI 192.168.206.32@tcp [8/512/0/180] [ 849.617686] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 849.621294] LNet: 53059:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 849.744622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 849.774554] LNet: 53078:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 849.782239] LNetError: 53078:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 849.789934] LNet: Removed LNI 10.1.2.3@tcp [ 849.792128] LNetError: 53078:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 849.815465] LNet: Removed LNI 192.168.206.32@tcp [ 849.863202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 849.944983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 849.969163] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 850.037814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 850.057306] LNet: 53102:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 850.064027] LNetError: 53102:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 850.080797] LNet: Removed LNI 192.168.206.32@tcp [ 850.099702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 850.164178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 850.185622] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 850.194315] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 850.198258] LNet: 53110:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 850.295312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 850.313384] LNet: 53128:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 850.317939] LNetError: 53128:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 850.323965] LNet: Removed LNI 10.1.2.3@tcp [ 850.325571] LNetError: 53128:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 850.351499] LNet: Removed LNI 192.168.206.32@tcp [ 850.377863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 850.453722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 850.480317] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 850.484182] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 850.486869] LNet: 53136:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 850.721810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 850.754959] LNet: 53162:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 850.770499] LNetError: 53162:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 850.796269] LNet: Removed LNI 10.1.2.3@tcp [ 850.809538] LNetError: 53162:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 850.828383] LNet: Removed LNI 192.168.206.32@tcp [ 850.851223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 850.904313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 850.924668] LNet: Added LNI 192.168.206.32@tcp [8/256/0/182] [ 850.929231] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 851.006802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 851.026993] LNet: 53191:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 851.034897] LNetError: 53191:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 851.042323] LNet: Removed LNI 10.1.2.3@tcp [ 851.045213] LNetError: 53191:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 851.059787] LNet: Removed LNI 192.168.206.32@tcp [ 851.083398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 851.164959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 851.191853] LNet: Added LNI 192.168.206.32@tcp [8/256/0/182] [ 851.195946] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 851.197985] LNet: 53199:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 851.300029] LNet: Removed LNI 10.1.2.3@tcp [ 851.312426] LNet: Removed LNI 192.168.206.32@tcp [ 851.926856] Key type .llcrypt unregistered [ 851.929560] Key type ._llcrypt unregistered [ 861.551194] Key type ._llcrypt registered [ 861.553712] Key type .llcrypt registered [ 862.742479] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 23:06:03 (1788836763) [ 864.108174] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 865.389697] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 23:06:06 (1788836766) [ 865.762456] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 865.777535] alg: No test for adler32 (adler32-zlib) [ 866.723758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 866.765659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 866.788617] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 867.482697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.206.32 [ 867.496970] LNet: 54460:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 867.506385] LNet: Removed LNI 192.168.206.32@tcp [ 872.803648] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 23:06:13 (1788836773) [ 872.900286] LNet: 55022:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 872.917395] LNetError: 55022:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 872.944155] LNet: Removed LNI 192.168.206.32@tcp [ 873.511391] Key type .llcrypt unregistered [ 873.515405] Key type ._llcrypt unregistered [ 878.802195] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 879.042840] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 881.665558] Key type ._llcrypt registered [ 881.668548] Key type .llcrypt registered [ 881.817570] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 881.829574] alg: No test for adler32 (adler32-zlib) [ 883.301713] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 890.021732] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 23:06:30 (1788836790) [ 890.192692] LNet: 55939:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 890.216464] LNetError: 55939:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 890.242331] LNet: Removed LNI 10.1.2.3@tcp [ 890.896363] Key type .llcrypt unregistered [ 890.898528] Key type ._llcrypt unregistered [ 897.214921] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 897.415156] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 897.419504] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 897.425443] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 900.203651] Key type ._llcrypt registered [ 900.205782] Key type .llcrypt registered [ 900.452230] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 900.470824] alg: No test for adler32 (adler32-zlib) [ 901.407078] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 907.896270] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 23:06:48 (1788836808) [ 909.575363] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 911.205435] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 23:06:52 (1788836812) [ 911.450765] LNet: 57124:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 911.459947] LNetError: 57124:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 911.472222] LNet: Removed LNI 10.1.2.3@tcp [ 911.993611] Key type .llcrypt unregistered [ 911.995658] Key type ._llcrypt unregistered [ 918.750829] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 919.017195] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 919.025853] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 919.037906] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 921.967935] Key type ._llcrypt registered [ 921.970377] Key type .llcrypt registered [ 922.152470] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 922.168331] alg: No test for adler32 (adler32-zlib) [ 923.146344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 923.226628] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 931.862752] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 23:07:12 (1788836832) [ 932.309483] LNet: 58350:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 932.319531] LNetError: 58350:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 932.335618] LNet: Removed LNI 10.1.2.3@tcp [ 932.903767] Key type .llcrypt unregistered [ 932.906387] Key type ._llcrypt unregistered [ 938.788542] Key type ._llcrypt registered [ 938.792157] Key type .llcrypt registered [ 939.007638] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 939.024658] alg: No test for adler32 (adler32-zlib) [ 939.971363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 940.006441] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 947.613627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.132@tcp [ 948.255646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.32@tcp -d 192.168.206.132@tcp -m GET -e local_interrupt -r 1 [ 948.285139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.32@tcp -d 192.168.206.32@tcp -m GET -e local_interrupt -r 1 [ 948.314590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.132@tcp [ 948.329797] LNet: There was an unexpected network error while writing to 192.168.206.132: rc = -22 [ 948.508249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 948.545563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 948.580558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 948.610827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 948.636662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 948.657869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 948.688442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 948.744535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 948.780849] LNet: 58784:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 948.806291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 948.861761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 948.906459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 948.931765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 948.957479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 949.008166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 949.059250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 949.635856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.32@tcp -d 192.168.206.132@tcp -m GET -e local_dropped -r 1 [ 949.685560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.32@tcp -d 192.168.206.32@tcp -m GET -e local_dropped -r 1 [ 949.740767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.132@tcp [ 949.766656] LNet: There was an unexpected network error while writing to 192.168.206.132: rc = -22 [ 950.029483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 950.064673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 950.095704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 950.122119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 950.153740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 950.198678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 950.254768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 950.304109] LNet: 1 local NIs in recovery (showing 1): 192.168.206.32@tcp [ 950.304907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 950.341328] LNet: 58894:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 950.366195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 950.399367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 950.432511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 950.462807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 950.501807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 950.540326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 950.574522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 951.069542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.32@tcp -d 192.168.206.132@tcp -m GET -e local_aborted -r 1 [ 951.112465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.32@tcp -d 192.168.206.32@tcp -m GET -e local_aborted -r 1 [ 951.162639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.132@tcp [ 951.197544] LNet: There was an unexpected network error while writing to 192.168.206.132: rc = -22 [ 951.205909] LNet: Skipped 1 previous similar message [ 951.409562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 951.443196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 951.477361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 951.515191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 951.538147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 951.563427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 951.595287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 951.643295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 951.662974] LNet: 59003:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 951.714618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 951.756389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 951.789061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 951.821375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 951.846460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 951.889087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 951.917224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 952.380466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.32@tcp -d 192.168.206.132@tcp -m GET -e local_no_route -r 1 [ 952.411791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.32@tcp -d 192.168.206.32@tcp -m GET -e local_no_route -r 1 [ 952.449132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.132@tcp [ 952.641580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 952.682707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 952.723643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 952.792563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 952.850363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 952.887648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 952.923700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 952.965432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 953.016062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 953.061469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 953.088298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 953.126585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 953.159766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 953.210704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 953.244149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 953.725621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.32@tcp -d 192.168.206.132@tcp -m GET -e local_timeout -r 1 [ 953.758716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.32@tcp -d 192.168.206.32@tcp -m GET -e local_timeout -r 1 [ 953.796603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.132@tcp [ 953.846713] LNet: There was an unexpected network error while writing to 192.168.206.132: rc = -22 [ 953.854963] LNet: Skipped 1 previous similar message [ 954.079160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 954.115287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 954.149586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 954.186799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 954.229767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 954.275302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 954.310936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 954.350916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 954.375558] LNet: 59221:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 954.381445] LNet: 59221:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 954.404057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 954.454812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 954.495216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 954.541183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 954.578244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 954.612973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 954.647637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 955.220201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.32@tcp -d 192.168.206.132@tcp -m GET -e local_error -r 1 [ 955.260829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.32@tcp -d 192.168.206.32@tcp -m GET -e local_error -r 1 [ 955.290811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.132@tcp [ 955.480688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 955.508652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 955.539195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 955.563824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 955.584596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 955.609800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 955.646947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 955.675306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 955.702559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 955.735092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 955.784729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 955.830262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 955.866393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 955.896482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 955.926747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 956.150945] LNet: 59371:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 956.161120] LNetError: 59371:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 956.184124] LNet: Removed LNI 192.168.206.32@tcp [ 956.533540] Key type .llcrypt unregistered [ 956.536796] Key type ._llcrypt unregistered [ 971.174897] Key type ._llcrypt registered [ 971.176225] Key type .llcrypt registered [ 972.179303] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 23:07:53 (1788836873) [ 972.975249] Key type .llcrypt unregistered [ 972.976601] Key type ._llcrypt unregistered [ 978.177106] Key type ._llcrypt registered [ 978.179409] Key type .llcrypt registered [ 978.359725] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 978.387789] alg: No test for adler32 (adler32-zlib) [ 979.281959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 979.300379] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 986.723773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.132@tcp [ 996.596703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 996.637978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 996.660733] LNet: Added LNI 192.168.206.32@tcp1 [8/256/0/180] [ 997.947898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.32@tcp -d 192.168.206.132@tcp -m GET -e local_interrupt -r 1 [ 997.978810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.32@tcp -d 192.168.206.132@tcp1 -m GET -e local_interrupt -r 1 [ 998.023736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.32@tcp -d 192.168.206.32@tcp -m GET -e local_interrupt -r 1 [ 998.075949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.32@tcp -d 192.168.206.32@tcp1 -m GET -e local_interrupt -r 1 [ 998.134621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.32@tcp1 -d 192.168.206.132@tcp -m GET -e local_interrupt -r 1 [ 998.199817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.32@tcp1 -d 192.168.206.132@tcp1 -m GET -e local_interrupt -r 1 [ 998.257806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.32@tcp1 -d 192.168.206.32@tcp -m GET -e local_interrupt -r 1 [ 998.294919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.32@tcp1 -d 192.168.206.32@tcp1 -m GET -e local_interrupt -r 1 [ 998.327496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.132@tcp [ 998.345516] LNet: There was an unexpected network error while writing to 192.168.206.132: rc = -22 [ 998.353168] LNet: 1 local NIs in recovery (showing 1): 192.168.206.32@tcp [ 998.558770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 998.587294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 998.616345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 998.657473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 998.709270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 998.744513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 998.794251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 998.835840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 998.855554] LNet: 60833:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 998.870412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 998.890141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 998.910892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 998.932692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 998.953762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 998.985812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 999.015498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 999.337944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.32@tcp -d 192.168.206.132@tcp -m GET -e local_dropped -r 1 [ 999.376148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.32@tcp -d 192.168.206.132@tcp1 -m GET -e local_dropped -r 1 [ 999.429510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.32@tcp -d 192.168.206.32@tcp -m GET -e local_dropped -r 1 [ 999.467410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.32@tcp -d 192.168.206.32@tcp1 -m GET -e local_dropped -r 1 [ 999.496352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.32@tcp1 -d 192.168.206.132@tcp -m GET -e local_dropped -r 1 [ 999.532320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.32@tcp1 -d 192.168.206.132@tcp1 -m GET -e local_dropped -r 1 [ 999.566084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.32@tcp1 -d 192.168.206.32@tcp -m GET -e local_dropped -r 1 [ 999.614985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.32@tcp1 -d 192.168.206.32@tcp1 -m GET -e local_dropped -r 1 [ 999.660994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.132@tcp [ 999.697125] LNet: There was an unexpected network error while writing to 192.168.206.132: rc = -22 [ 999.707589] LNet: Skipped 2 previous similar messages [ 999.938816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 999.965986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1000.003706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1000.046237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1000.076748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1000.112923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1000.136633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1000.156553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1000.167909] LNet: 60954:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1000.183324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1000.222840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1000.240898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1000.273342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1000.300387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1000.343750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1000.384247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1000.796453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.32@tcp -d 192.168.206.132@tcp -m GET -e local_aborted -r 1 [ 1000.832857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.32@tcp -d 192.168.206.132@tcp1 -m GET -e local_aborted -r 1 [ 1000.855923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.32@tcp -d 192.168.206.32@tcp -m GET -e local_aborted -r 1 [ 1000.886814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.32@tcp -d 192.168.206.32@tcp1 -m GET -e local_aborted -r 1 [ 1000.919325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.32@tcp1 -d 192.168.206.132@tcp -m GET -e local_aborted -r 1 [ 1000.956327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.32@tcp1 -d 192.168.206.132@tcp1 -m GET -e local_aborted -r 1 [ 1000.985349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.32@tcp1 -d 192.168.206.32@tcp -m GET -e local_aborted -r 1 [ 1001.021996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.32@tcp1 -d 192.168.206.32@tcp1 -m GET -e local_aborted -r 1 [ 1001.057592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.132@tcp [ 1001.081037] LNet: There was an unexpected network error while writing to 192.168.206.132: rc = -22 [ 1001.089371] LNet: Skipped 2 previous similar messages [ 1001.294721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1001.334285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1001.366459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1001.398718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1001.420795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1001.440916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1001.460216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1001.486971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1001.506787] LNet: 61075:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1001.521922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1001.548739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1001.571405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1001.593454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1001.635371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1001.678801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1001.715298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1002.059903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.32@tcp -d 192.168.206.132@tcp -m GET -e local_no_route -r 1 [ 1002.097755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.32@tcp -d 192.168.206.132@tcp1 -m GET -e local_no_route -r 1 [ 1002.137932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.32@tcp -d 192.168.206.32@tcp -m GET -e local_no_route -r 1 [ 1002.185894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.32@tcp -d 192.168.206.32@tcp1 -m GET -e local_no_route -r 1 [ 1002.211539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.32@tcp1 -d 192.168.206.132@tcp -m GET -e local_no_route -r 1 [ 1002.239101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.32@tcp1 -d 192.168.206.132@tcp1 -m GET -e local_no_route -r 1 [ 1002.258615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.32@tcp1 -d 192.168.206.32@tcp -m GET -e local_no_route -r 1 [ 1002.282916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.32@tcp1 -d 192.168.206.32@tcp1 -m GET -e local_no_route -r 1 [ 1002.309965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.132@tcp [ 1002.503111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1002.545944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1002.573921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1002.601825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1002.636855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1002.677087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1002.713402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1002.743297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1002.764146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1002.807700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1002.855340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1002.885220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1002.913415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1002.943184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1002.965351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1003.325443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.32@tcp -d 192.168.206.132@tcp -m GET -e local_timeout -r 1 [ 1003.346296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.32@tcp -d 192.168.206.132@tcp1 -m GET -e local_timeout -r 1 [ 1003.366411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.32@tcp -d 192.168.206.32@tcp -m GET -e local_timeout -r 1 [ 1003.387918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.32@tcp -d 192.168.206.32@tcp1 -m GET -e local_timeout -r 1 [ 1003.407060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.32@tcp1 -d 192.168.206.132@tcp -m GET -e local_timeout -r 1 [ 1003.433135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.32@tcp1 -d 192.168.206.132@tcp1 -m GET -e local_timeout -r 1 [ 1003.469501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.32@tcp1 -d 192.168.206.32@tcp -m GET -e local_timeout -r 1 [ 1003.502432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.32@tcp1 -d 192.168.206.32@tcp1 -m GET -e local_timeout -r 1 [ 1003.527975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.132@tcp [ 1003.539838] LNet: There was an unexpected network error while writing to 192.168.206.132: rc = -22 [ 1003.543811] LNet: Skipped 5 previous similar messages [ 1003.682356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1003.719571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1003.758675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1003.808530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1003.851867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1003.882798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1003.904525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1003.922080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1003.932663] LNet: 61318:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1003.937629] LNet: 61318:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1003.945848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1003.967361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1003.990435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1004.011827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1004.039498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1004.077260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1004.098417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1004.439484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.32@tcp -d 192.168.206.132@tcp -m GET -e local_error -r 1 [ 1004.459528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.32@tcp -d 192.168.206.132@tcp1 -m GET -e local_error -r 1 [ 1004.480342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.32@tcp -d 192.168.206.32@tcp -m GET -e local_error -r 1 [ 1004.507725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.32@tcp -d 192.168.206.32@tcp1 -m GET -e local_error -r 1 [ 1004.529749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.32@tcp1 -d 192.168.206.132@tcp -m GET -e local_error -r 1 [ 1004.571939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.32@tcp1 -d 192.168.206.132@tcp1 -m GET -e local_error -r 1 [ 1004.611784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.32@tcp1 -d 192.168.206.32@tcp -m GET -e local_error -r 1 [ 1004.644553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.32@tcp1 -d 192.168.206.32@tcp1 -m GET -e local_error -r 1 [ 1004.676593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.132@tcp [ 1004.914584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1004.939794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1004.981169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1005.013314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1005.040559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1005.097297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1005.132797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1005.167058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1005.223357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1005.285366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1005.338963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1005.373650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1005.412333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1005.464117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1005.488883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1009.829978] LNet: 61515:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1009.835816] LNetError: 61515:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1009.840752] LNet: Removed LNI 192.168.206.32@tcp [ 1009.854222] LNet: Removed LNI 192.168.206.32@tcp1 [ 1010.165529] Key type .llcrypt unregistered [ 1010.168625] Key type ._llcrypt unregistered [ 1025.085884] Key type ._llcrypt registered [ 1025.088950] Key type .llcrypt registered [ 1026.463527] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 23:08:47 (1788836927) [ 1027.207834] Key type .llcrypt unregistered [ 1027.210151] Key type ._llcrypt unregistered [ 1032.183668] Key type ._llcrypt registered [ 1032.185695] Key type .llcrypt registered [ 1032.333565] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1032.349285] alg: No test for adler32 (adler32-zlib) [ 1033.286485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1033.310079] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 1039.742035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.132@tcp [ 1040.322990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.32@tcp -d 192.168.206.132@tcp -m GET -e remote_dropped -r 1 [ 1040.359066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.32@tcp -d 192.168.206.32@tcp -m GET -e remote_dropped -r 1 [ 1040.389766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.132@tcp [ 1040.407567] LNet: There was an unexpected network error while writing to 192.168.206.132: rc = -22 [ 1040.584185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1040.608217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1040.635853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1040.655305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1040.675555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1040.696066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1040.736390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1040.779127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1040.796104] LNet: 62760:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1040.813480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1040.843868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1040.878246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1040.913995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1040.944750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1040.989645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1041.024458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1041.475216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.32@tcp -d 192.168.206.132@tcp -m GET -e remote_error -r 1 [ 1041.507629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.32@tcp -d 192.168.206.32@tcp -m GET -e remote_error -r 1 [ 1041.543903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.132@tcp [ 1041.562479] LNet: There was an unexpected network error while writing to 192.168.206.132: rc = -22 [ 1041.755357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1041.781842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1041.810658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1041.861689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1041.899081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1041.935192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1041.975254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1042.012207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1042.030668] LNet: 62869:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1042.065521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1042.097522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1042.137643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1042.178705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1042.212971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1042.248173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1042.280637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1042.813269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.32@tcp -d 192.168.206.132@tcp -m GET -e remote_timeout -r 1 [ 1042.849377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.32@tcp -d 192.168.206.32@tcp -m GET -e remote_timeout -r 1 [ 1042.887764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.132@tcp [ 1042.913917] LNet: There was an unexpected network error while writing to 192.168.206.132: rc = -22 [ 1043.127691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1043.151175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1043.173570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1043.196682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1043.220541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1043.257751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1043.289879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1043.318159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1043.331816] LNet: 62978:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1043.347499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1043.377060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1043.415069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1043.457974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1043.490425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1043.527762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1043.560204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1043.798758] LNet: 63019:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1043.805210] LNetError: 63019:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1043.826878] LNet: Removed LNI 192.168.206.32@tcp [ 1044.174609] Key type .llcrypt unregistered [ 1044.177216] Key type ._llcrypt unregistered [ 1059.122241] Key type ._llcrypt registered [ 1059.127138] Key type .llcrypt registered [ 1060.143952] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 23:09:21 (1788836961) [ 1060.827081] Key type .llcrypt unregistered [ 1060.833296] Key type ._llcrypt unregistered [ 1065.772962] Key type ._llcrypt registered [ 1065.776690] Key type .llcrypt registered [ 1065.932033] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1065.951978] alg: No test for adler32 (adler32-zlib) [ 1066.832402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1066.855782] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 1073.274731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.132@tcp [ 1081.310636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1081.370579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1081.418830] LNet: Added LNI 192.168.206.32@tcp1 [8/256/0/180] [ 1082.955798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.32@tcp -d 192.168.206.132@tcp -m GET -e remote_dropped -r 1 [ 1083.031234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.32@tcp -d 192.168.206.132@tcp1 -m GET -e remote_dropped -r 1 [ 1083.064962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.32@tcp -d 192.168.206.32@tcp -m GET -e remote_dropped -r 1 [ 1083.091507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.32@tcp -d 192.168.206.32@tcp1 -m GET -e remote_dropped -r 1 [ 1083.115220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.32@tcp1 -d 192.168.206.132@tcp -m GET -e remote_dropped -r 1 [ 1083.182961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.32@tcp1 -d 192.168.206.132@tcp1 -m GET -e remote_dropped -r 1 [ 1083.261984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.32@tcp1 -d 192.168.206.32@tcp -m GET -e remote_dropped -r 1 [ 1083.317715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.32@tcp1 -d 192.168.206.32@tcp1 -m GET -e remote_dropped -r 1 [ 1083.369690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.132@tcp [ 1083.407067] LNet: There was an unexpected network error while writing to 192.168.206.132: rc = -22 [ 1083.414951] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.132@tcp [ 1083.712905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1083.774908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1083.824851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1083.892470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1083.948962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1083.994605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1084.046454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1084.075790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1084.089788] LNet: 64479:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1084.105047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1084.154254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1084.202970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1084.250972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1084.307397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1084.362339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1084.411634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1084.454508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1085.002767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.32@tcp -d 192.168.206.132@tcp -m GET -e remote_error -r 1 [ 1085.045698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.32@tcp -d 192.168.206.132@tcp1 -m GET -e remote_error -r 1 [ 1085.104203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.32@tcp -d 192.168.206.32@tcp -m GET -e remote_error -r 1 [ 1085.167934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.32@tcp -d 192.168.206.32@tcp1 -m GET -e remote_error -r 1 [ 1085.238560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.32@tcp1 -d 192.168.206.132@tcp -m GET -e remote_error -r 1 [ 1085.278501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.32@tcp1 -d 192.168.206.132@tcp1 -m GET -e remote_error -r 1 [ 1085.317748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.32@tcp1 -d 192.168.206.32@tcp -m GET -e remote_error -r 1 [ 1085.345956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.32@tcp1 -d 192.168.206.32@tcp1 -m GET -e remote_error -r 1 [ 1085.396375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.132@tcp [ 1085.418921] LNet: There was an unexpected network error while writing to 192.168.206.132: rc = -22 [ 1085.423470] LNet: Skipped 2 previous similar messages [ 1085.684665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1085.741976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1085.778823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1085.808347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1085.870425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1085.935805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1086.038388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1086.095333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1086.134338] LNet: 64603:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1086.160197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1086.201446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1086.235588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1086.271534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1086.325620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1086.360110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1086.390730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1086.424201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1087.198911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.32@tcp -d 192.168.206.132@tcp -m GET -e remote_timeout -r 1 [ 1087.241045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.32@tcp -d 192.168.206.132@tcp1 -m GET -e remote_timeout -r 1 [ 1087.291691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.32@tcp -d 192.168.206.32@tcp -m GET -e remote_timeout -r 1 [ 1087.332987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.32@tcp -d 192.168.206.32@tcp1 -m GET -e remote_timeout -r 1 [ 1087.401240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.32@tcp1 -d 192.168.206.132@tcp -m GET -e remote_timeout -r 1 [ 1087.460954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.32@tcp1 -d 192.168.206.132@tcp1 -m GET -e remote_timeout -r 1 [ 1087.508791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.32@tcp1 -d 192.168.206.32@tcp -m GET -e remote_timeout -r 1 [ 1087.536992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.32@tcp1 -d 192.168.206.32@tcp1 -m GET -e remote_timeout -r 1 [ 1087.562392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.132@tcp [ 1087.576442] LNet: There was an unexpected network error while writing to 192.168.206.132: rc = -22 [ 1087.764462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1087.792371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1087.831715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1087.858643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1087.881753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1087.903542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1087.929450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1087.972677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1088.001307] LNet: 64726:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1088.021946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1088.052940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1088.110585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1088.137818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1088.159603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1088.179956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1088.203738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1088.239333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1093.706342] LNet: 64804:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1093.716570] LNetError: 64804:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1093.726367] LNet: Removed LNI 192.168.206.32@tcp [ 1093.737853] LNet: Removed LNI 192.168.206.32@tcp1 [ 1094.229333] Key type .llcrypt unregistered [ 1094.239325] Key type ._llcrypt unregistered [ 1107.976726] Key type ._llcrypt registered [ 1107.980095] Key type .llcrypt registered [ 1109.461668] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 23:10:10 (1788837010) [ 1110.903343] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1112.424959] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 23:10:13 (1788837013) [ 1113.341518] Key type .llcrypt unregistered [ 1113.344954] Key type ._llcrypt unregistered [ 1118.579450] Key type ._llcrypt registered [ 1118.581818] Key type .llcrypt registered [ 1118.814230] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1118.842708] alg: No test for adler32 (adler32-zlib) [ 1119.744189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1119.786625] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 1126.349747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.132@tcp [ 1126.997863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.32@tcp -d 192.168.206.132@tcp -m GET -e network_timeout -r 1 [ 1127.062768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.32@tcp -d 192.168.206.32@tcp -m GET -e network_timeout -r 1 [ 1127.118478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.132@tcp [ 1127.137967] LNet: There was an unexpected network error while writing to 192.168.206.132: rc = -22 [ 1127.326389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1127.371308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1127.410844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1127.445298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1127.473667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1127.508769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1127.540561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1127.572517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1127.594472] LNet: 66343:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1127.614094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1127.643527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1127.672305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1127.699792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1127.749513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1127.820934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1128.156565] LNet: 66382:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1128.174879] LNetError: 66382:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1129.253905] LNet: Removed LNI 192.168.206.32@tcp [ 1129.594450] Key type .llcrypt unregistered [ 1129.597539] Key type ._llcrypt unregistered [ 1146.893630] Key type ._llcrypt registered [ 1146.896258] Key type .llcrypt registered [ 1147.186903] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1147.204226] alg: No test for adler32 (adler32-zlib) [ 1148.127606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1148.167265] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 1154.730203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.132@tcp [ 1164.272416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1164.309972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1164.337402] LNet: Added LNI 192.168.206.32@tcp1 [8/256/0/180] [ 1165.564417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.32@tcp -d 192.168.206.132@tcp -m GET -e network_timeout -r 1 [ 1165.605204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.32@tcp -d 192.168.206.132@tcp1 -m GET -e network_timeout -r 1 [ 1165.642456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.32@tcp -d 192.168.206.32@tcp -m GET -e network_timeout -r 1 [ 1165.678235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.32@tcp -d 192.168.206.32@tcp1 -m GET -e network_timeout -r 1 [ 1165.729733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.32@tcp1 -d 192.168.206.132@tcp -m GET -e network_timeout -r 1 [ 1165.804787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.32@tcp1 -d 192.168.206.132@tcp1 -m GET -e network_timeout -r 1 [ 1165.841581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.32@tcp1 -d 192.168.206.32@tcp -m GET -e network_timeout -r 1 [ 1165.893519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.32@tcp1 -d 192.168.206.32@tcp1 -m GET -e network_timeout -r 1 [ 1165.930531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.132@tcp [ 1165.948929] LNet: There was an unexpected network error while writing to 192.168.206.132: rc = -22 [ 1166.189975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1166.228502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1166.260262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1166.303099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1166.361427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1166.417546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1166.486561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1166.549393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1166.598282] LNet: 67258:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1166.625956] LNet: 1 local NIs in recovery (showing 1): 192.168.206.32@tcp [ 1166.627139] LNet: There was an unexpected network error while writing to 192.168.206.132: rc = -22 [ 1166.630568] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.132@tcp [ 1166.633254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1166.671665] LNet: Skipped 1 previous similar message [ 1166.686293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1166.734383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1166.766327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1166.789863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1166.815693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1170.750576] LNet: There was an unexpected network error while writing to 192.168.206.132: rc = -22 [ 1171.011377] LNet: 67332:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1171.017098] LNetError: 67332:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1171.022375] LNet: Removed LNI 192.168.206.32@tcp [ 1171.039973] LNet: Removed LNI 192.168.206.32@tcp1 [ 1171.487575] Key type .llcrypt unregistered [ 1171.489335] Key type ._llcrypt unregistered [ 1188.533569] Key type ._llcrypt registered [ 1188.540946] Key type .llcrypt registered [ 1189.766990] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 23:11:30 (1788837090) [ 1190.190090] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1190.201772] alg: No test for adler32 (adler32-zlib) [ 1191.115525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1191.166859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1191.187035] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 1191.842694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1191.855665] LNet: Added LNI 192.168.206.32@tcp1 [8/256/0/180] [ 1192.719870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.32@tcp [ 1192.775155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1192.806748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1192.837188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1192.869177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1192.897428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1192.930454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.32@tcp [ 1192.947357] LNetError: 68430:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.206.32@tcp to recovery [ 1193.183284] LNet: 1 local NIs in recovery (showing 1): 192.168.206.32@tcp [ 1193.184349] LNet: There was an unexpected network error while writing to 192.168.206.32: rc = -22 [ 1195.231349] LNet: There was an unexpected network error while writing to 192.168.206.32: rc = -22 [ 1196.432408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1199.328855] LNet: There was an unexpected network error while writing to 192.168.206.32: rc = -22 [ 1199.914591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1199.966448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1200.024535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1200.039752] LNet: 68537:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1200.045823] LNetError: 68537:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1201.055290] LNet: Removed LNI 192.168.206.32@tcp [ 1201.061831] LNetError: 68537:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1201.110097] LNet: Removed LNI 192.168.206.32@tcp1 [ 1201.153978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1201.216265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1201.232458] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 1201.784598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1201.801346] LNet: Added LNI 192.168.206.32@tcp1 [8/256/0/180] [ 1202.461468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.32@tcp [ 1202.492859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1202.541941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1202.580447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1202.611841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1202.647276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1202.682407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1202.720966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.32@tcp [ 1202.768067] LNetError: 68810:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.206.32@tcp to recovery [ 1203.232310] LNet: There was an unexpected network error while writing to 192.168.206.32: rc = -22 [ 1206.237917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1209.379388] LNet: There was an unexpected network error while writing to 192.168.206.32: rc = -22 [ 1209.386749] LNet: Skipped 1 previous similar message [ 1214.021163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1214.047794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1214.083855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1219.149536] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 23:12:00 (1788837120) [ 1219.231981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1219.255285] LNet: 69520:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1219.265728] LNetError: 69520:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1219.273664] LNet: Removed LNI 192.168.206.32@tcp [ 1219.284794] LNet: Removed LNI 192.168.206.32@tcp1 [ 1219.312576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1219.353217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1219.384139] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 1219.875496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1219.893372] LNet: Added LNI 192.168.206.32@tcp1 [8/256/0/180] [ 1220.679668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.32@tcp [ 1220.750667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1220.780689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1220.828532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1220.855878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1220.904600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1220.949462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.32@tcp --health 0 [ 1221.025930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1221.407247] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.32@tcp [ 1221.407514] LNet: There was an unexpected network error while writing to 192.168.206.32: rc = -22 [ 1221.429509] LNet: Skipped 1 previous similar message [ 1236.329679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1236.458968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1236.482534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1236.504961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.32@tcp --health 500 [ 1236.586562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1243.038599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1243.093488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1243.109250] LNet: 70020:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1243.115687] LNetError: 70020:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1243.123202] LNetError: 70020:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1243.130713] LNet: Removed LNI 192.168.206.32@tcp [ 1243.144411] LNet: Removed LNI 192.168.206.32@tcp1 [ 1243.158091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1243.191816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1243.206950] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 1243.709342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1243.726617] LNet: Added LNI 192.168.206.32@tcp1 [8/256/0/180] [ 1244.356226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.32@tcp [ 1244.380979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1244.430141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1244.473243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1244.506199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1244.548122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1244.586662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1244.628547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.32@tcp --health 0 [ 1245.215336] LNet: There was an unexpected network error while writing to 192.168.206.32: rc = -22 [ 1245.224428] LNet: Skipped 3 previous similar messages [ 1245.889821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1256.011334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1256.048269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1256.072529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1262.026325] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 23:12:43 (1788837163) [ 1262.586768] LNet: 71066:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1262.591743] LNetError: 71066:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1262.595902] LNetError: 71066:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1262.600793] LNet: Removed LNI 192.168.206.32@tcp [ 1262.616778] LNet: Removed LNI 192.168.206.32@tcp1 [ 1263.219286] Key type .llcrypt unregistered [ 1263.224357] Key type ._llcrypt unregistered [ 1269.065703] Key type ._llcrypt registered [ 1269.068441] Key type .llcrypt registered [ 1269.238803] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1269.250890] alg: No test for adler32 (adler32-zlib) [ 1270.136925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1270.186927] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 1278.239876] Lustre: DEBUG MARKER: Initial discovery [ 1278.448415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.132@tcp [ 1280.480986] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1280.705886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1280.758934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.132@tcp [ 1280.806693] LNet: There was an unexpected network error while writing to 192.168.206.132: rc = -22 [ 1280.832589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1280.878074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.206.132@tcp -r 1 -m GET -l 3 [ 1280.939886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.206.132@tcp -r 1 -m GET -e local_error [ 1280.990921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.206.132@tcp -r 1 -m PUT -l 6 [ 1281.516693] LNet: 1 local NIs in recovery (showing 1): 192.168.206.32@tcp [ 1282.265081] Lustre: DEBUG MARKER: Force oleg632-client.virtnet to discover 192.168.206.132@tcp (in background) [ 1282.341311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.132@tcp [ 1283.628863] Lustre: DEBUG MARKER: Force oleg632-server to discover 192.168.206.32@tcp [ 1285.407740] LNet: There was an unexpected network error while writing to 192.168.206.132: rc = -22 [ 1285.691127] Lustre: DEBUG MARKER: Wait for 71817 [ 1287.154921] Lustre: DEBUG MARKER: Finished wait on 71817 [ 1287.258080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1287.298407] LNet: There was an unexpected network error while writing to 192.168.206.132: rc = -22 [ 1287.309847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1287.553615] LNet: 72275:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1287.559456] LNetError: 72275:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1287.571250] LNet: Removed LNI 192.168.206.32@tcp [ 1287.921152] Key type .llcrypt unregistered [ 1287.930174] Key type ._llcrypt unregistered [ 1304.295238] Key type ._llcrypt registered [ 1304.297393] Key type .llcrypt registered [ 1305.389485] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 23:13:26 (1788837206) [ 1305.885249] Key type .llcrypt unregistered [ 1305.886865] Key type ._llcrypt unregistered [ 1311.977012] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1312.094447] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1312.101887] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1312.113381] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1314.608392] Key type ._llcrypt registered [ 1314.612719] Key type .llcrypt registered [ 1314.805404] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1314.825614] alg: No test for adler32 (adler32-zlib) [ 1315.720257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1315.753675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1315.776278] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 1316.388231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1316.412383] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1323.249318] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 23:13:44 (1788837224) [ 1323.496088] LNet: 74344:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1323.500638] LNetError: 74344:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1323.505814] LNet: Removed LNI 10.1.2.3@tcp [ 1323.519419] LNet: Removed LNI 192.168.206.32@tcp [ 1324.158125] Key type .llcrypt unregistered [ 1324.163597] Key type ._llcrypt unregistered [ 1330.163437] Key type ._llcrypt registered [ 1330.167871] Key type .llcrypt registered [ 1330.359487] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1330.377373] alg: No test for adler32 (adler32-zlib) [ 1331.269172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1331.343736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1331.360422] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 1332.034890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1332.055703] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1333.040649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.206.32@tcp [ 1334.118914] LNetError: Unexpected error -22 connecting to 192.168.206.32@tcp at host 192.168.206.32:988 [ 1338.929616] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 23:13:59 (1788837239) [ 1339.119303] LNet: 75472:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1339.129612] LNetError: 75472:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1340.193028] LNet: Removed LNI 10.1.2.3@tcp [ 1340.195175] LNetError: 75472:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1340.214900] LNet: Removed LNI 192.168.206.32@tcp [ 1340.853419] Key type .llcrypt unregistered [ 1340.856981] Key type ._llcrypt unregistered [ 1346.288955] Key type ._llcrypt registered [ 1346.291741] Key type .llcrypt registered [ 1346.542432] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1346.572296] alg: No test for adler32 (adler32-zlib) [ 1347.482979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1347.536967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1347.556551] LNet: Added LNI 192.168.206.32@tcp1 [8/256/0/180] [ 1348.328065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1348.347932] LNet: Added LNI 192.168.206.32@tcp2 [8/256/0/180] [ 1349.066905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.206.32@tcp1 --nid 192.168.206.32@tcp2 [ 1353.074909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.32@tcp1 192.168.206.32@tcp2 [ 1353.100927] LNetError: 76305:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.206.32@tcp1 to 192.168.206.32@tcp2 - Route table may be misconfigured [ 1353.122575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.32@tcp2 192.168.206.32@tcp1 [ 1358.673808] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 23:14:19 (1788837259) [ 1358.791305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1358.823195] LNet: 76870:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1358.832562] LNetError: 76870:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1358.845124] LNet: Removed LNI 192.168.206.32@tcp1 [ 1359.912630] LNet: Removed LNI 192.168.206.32@tcp2 [ 1359.970703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1360.071533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1360.104577] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 1360.785240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1360.817730] LNet: Added LNI 192.168.206.32@tcp1 [8/256/0/180] [ 1361.641288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.32@tcp [ 1361.685489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.32@tcp 192.168.206.32@tcp [ 1361.736266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.32@tcp1 192.168.206.32@tcp1 [ 1361.777442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.206.32@tcp -d 192.168.206.32@tcp -e network_timeout [ 1361.814459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.206.32@tcp -d 192.168.206.32@tcp1 -e network_timeout [ 1361.849065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.206.32@tcp1 -d 192.168.206.32@tcp -e network_timeout [ 1361.876424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.206.32@tcp1 -d 192.168.206.32@tcp1 -e network_timeout [ 1361.912295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.32@tcp [ 1361.955407] LNet: There was an unexpected network error while writing to 192.168.206.32: rc = -22 [ 1361.987665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1362.088905] LNet: 1 local NIs in recovery (showing 1): 192.168.206.32@tcp [ 1362.130866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1362.268279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1368.033745] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 23:14:28 (1788837268) [ 1368.127109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1368.156069] LNet: 77735:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1368.162573] LNetError: 77735:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1368.170994] LNetError: 77735:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1368.187166] LNet: Removed LNI 192.168.206.32@tcp [ 1368.213352] LNet: Removed LNI 192.168.206.32@tcp1 [ 1368.242414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1368.312482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1368.797099] Key type .llcrypt unregistered [ 1368.799236] Key type ._llcrypt unregistered [ 1378.046428] Key type ._llcrypt registered [ 1378.048839] Key type .llcrypt registered [ 1379.228139] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 23:14:40 (1788837280) [ 1379.650636] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1379.662643] alg: No test for adler32 (adler32-zlib) [ 1380.657515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1382.728591] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1384.660227] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 23:14:45 (1788837285) [ 1384.760773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1384.808023] LNet: 78866:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1384.868318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1385.043809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1385.099916] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 1386.174653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1386.230679] LNet: Added LNI 192.168.206.32@tcp1 [8/256/0/180] [ 1387.652803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.32@tcp [ 1387.751871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.32@tcp1 [ 1387.856768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.32@tcp1 [ 1397.040382] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 23:14:57 (1788837297) [ 1398.802654] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg632-server [ 1400.441849] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 23:15:01 (1788837301) [ 1402.490459] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg632-server [ 1404.215424] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 23:15:05 (1788837305) [ 1405.925383] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg632-server [ 1407.980660] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 23:15:08 (1788837308) [ 1409.864803] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg632-server [ 1411.916995] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 23:15:12 (1788837312) [ 1414.163043] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg632-server [ 1416.144794] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 23:15:16 (1788837316) [ 1418.456263] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg632-server [ 1420.428394] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 23:15:20 (1788837320) [ 1422.362141] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg632-server [ 1424.307766] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 23:15:24 (1788837324) [ 1426.216984] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg632-server [ 1427.751836] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 23:15:28 (1788837328) [ 1429.294402] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg632-server [ 1430.796062] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 23:15:31 (1788837331) [ 1430.993892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1431.016654] LNet: 82751:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1431.022232] LNetError: 82751:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1431.027377] LNet: Removed LNI 192.168.206.32@tcp [ 1432.104108] LNet: Removed LNI 192.168.206.32@tcp1 [ 1432.159472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1432.229812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1432.250065] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 1432.881855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1432.970885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.32@tcp [ 1433.161341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1433.197575] LNet: 82899:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1433.211511] LNetError: 82899:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1433.217811] LNetError: 82899:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1433.248282] LNet: Removed LNI 192.168.206.32@tcp [ 1433.292863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1433.377251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1433.422231] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 1434.210525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.32@tcp [ 1434.407391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1434.423545] LNetError: 83044:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1434.436395] LNet: Removed LNI 192.168.206.32@tcp [ 1434.469482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1434.538518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1434.558529] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 1435.211819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1435.298220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.32@tcp [ 1435.466361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1435.486498] LNet: 83191:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1435.500072] LNet: 83191:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1436.582851] LNet: Removed LNI 192.168.206.32@tcp [ 1436.611331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1436.668728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1436.690569] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 1437.386507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.32@tcp [ 1437.555556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1437.598686] LNetError: 83336:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1437.608786] LNetError: 83336:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1437.622492] LNet: Removed LNI 192.168.206.32@tcp [ 1437.644423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1437.703283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1437.727105] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 1438.326320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1438.397374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.32@tcp [ 1438.591107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1439.658241] LNet: Removed LNI 192.168.206.32@tcp [ 1439.698128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1439.759820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1439.794097] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 1440.565614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.32@tcp [ 1441.920529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1441.944472] LNet: 83638:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1441.950025] LNet: 83638:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1441.958540] LNetError: 83638:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1441.969567] LNetError: 83638:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1441.988855] LNet: Removed LNI 192.168.206.32@tcp [ 1442.006451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1442.049799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1442.077189] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 1442.671937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1442.738475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.32@tcp [ 1444.035642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1445.100808] LNet: Removed LNI 192.168.206.32@tcp [ 1445.144422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1445.239496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1445.277507] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 1446.043429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.32@tcp [ 1446.213273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1447.271079] LNet: Removed LNI 192.168.206.32@tcp [ 1447.301653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1447.388141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1447.447668] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 1448.146411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1448.241759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.32@tcp [ 1448.465034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1449.510135] LNet: Removed LNI 192.168.206.32@tcp [ 1449.549352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1449.647075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1449.674077] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 1450.345541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.32@tcp [ 1450.685520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1450.701512] LNet: 84232:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1450.709706] LNet: 84232:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1450.720349] LNetError: 84232:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1450.728661] LNetError: 84232:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1450.749531] LNet: Removed LNI 192.168.206.32@tcp [ 1450.774874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1450.828139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1450.850927] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 1451.613683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1451.675660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.32@tcp [ 1451.984976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1453.031701] LNet: Removed LNI 192.168.206.32@tcp [ 1453.088860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1453.155112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1453.179624] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 1454.071127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.32@tcp [ 1454.356808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1455.398645] LNet: Removed LNI 192.168.206.32@tcp [ 1455.435690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1455.513638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1455.543314] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 1456.176658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1456.239161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1456.291691] LNet: Removed LNI 192.168.206.32@tcp [ 1456.312996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1456.361838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1456.387129] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 1457.180539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1457.255464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1457.287652] LNet: Removed LNI 192.168.206.32@tcp [ 1457.300184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1457.373690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1457.480542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1457.522617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1457.585519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1457.682203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1457.734604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1457.785319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1457.871182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1457.945968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1458.027585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1458.071587] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 1458.899917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1458.969898] LNet: Removed LNI 192.168.206.32@tcp [ 1459.000868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1459.090038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1465.596136] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 23:16:06 (1788837366) [ 1465.798203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1465.860436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1465.931500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1465.971280] LNet: Added LNI 192.168.206.32@tcp231 [8/256/0/180] [ 1467.362052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1467.398264] LNet: 85685:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1467.402913] LNet: 85685:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 1467.414308] LNetError: 85685:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1467.418771] LNetError: 85685:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1467.439719] LNet: Removed LNI 192.168.206.32@tcp231 [ 1467.469546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1467.547982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1467.587845] LNet: Added LNI 192.168.206.32@tcp231 [8/256/0/180] [ 1467.595456] LNet: 85693:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1468.092820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1468.122820] LNet: Removed LNI 192.168.206.32@tcp231 [ 1468.139662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1468.170218] LNet: Added LNI 192.168.206.32@tcp231 [8/256/0/180] [ 1469.523528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1469.553217] LNet: Removed LNI 192.168.206.32@tcp231 [ 1469.584605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1469.666636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1469.702651] LNet: Added LNI 192.168.206.32@tcp231 [8/256/0/0] [ 1469.706925] LNet: 85845:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1470.150762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1470.179235] LNet: Removed LNI 192.168.206.32@tcp231 [ 1470.199264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1470.219961] LNet: Added LNI 192.168.206.32@tcp231 [8/256/0/0] [ 1471.357398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1471.421472] LNet: Removed LNI 192.168.206.32@tcp231 [ 1471.468459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1471.602091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1471.648360] LNet: Added LNI 192.168.206.32@tcp231 [8/256/0/60] [ 1471.655064] LNet: 85997:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1472.101838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1472.125785] LNet: Removed LNI 192.168.206.32@tcp231 [ 1472.142973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1472.167458] LNet: Added LNI 192.168.206.32@tcp231 [8/256/0/60] [ 1473.655852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1473.710479] LNet: Removed LNI 192.168.206.32@tcp231 [ 1473.730986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1473.850700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1473.883229] LNet: Added LNI 192.168.206.32@tcp231 [8/256/0/180] [ 1473.892231] LNet: 86148:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1474.317756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1474.351850] LNet: Removed LNI 192.168.206.32@tcp231 [ 1474.372415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1474.392822] LNet: Added LNI 192.168.206.32@tcp231 [8/256/0/180] [ 1482.527771] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 23:16:23 (1788837383) [ 1482.651218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1482.693554] LNet: Removed LNI 192.168.206.32@tcp231 [ 1482.736969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1482.807821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1482.834225] LNet: Added LNI 192.168.206.32@tcp232 [8/256/0/180] [ 1483.807383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1483.862845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1483.898361] LNet: Removed LNI 192.168.206.32@tcp232 [ 1483.922280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1483.965660] LNet: Added LNI 192.168.206.32@tcp232 [8/256/0/180] [ 1484.702621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1484.773853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1484.822322] LNet: Removed LNI 192.168.206.32@tcp232 [ 1484.853617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1484.907133] LNet: Added LNI 192.168.206.32@tcp232 [8/256/0/180] [ 1485.725842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1485.758805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1485.799203] LNet: Removed LNI 192.168.206.32@tcp232 [ 1485.832208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1485.876639] LNet: Added LNI 192.168.206.32@tcp232 [8/256/0/180] [ 1486.664361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1486.716837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1486.753289] LNet: Removed LNI 192.168.206.32@tcp232 [ 1486.783390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1493.553900] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 23:16:34 (1788837394) [ 1494.801561] Key type .llcrypt unregistered [ 1494.804682] Key type ._llcrypt unregistered [ 1502.299690] Key type ._llcrypt registered [ 1502.308534] Key type .llcrypt registered [ 1502.593910] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1502.618244] alg: No test for adler32 (adler32-zlib) [ 1503.550424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1503.573854] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 1510.933657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.132@tcp [ 1519.091367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1519.140348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1519.179823] LNet: Added LNI 192.168.206.32@tcp1 [8/256/0/180] [ 1520.678446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.32@tcp -d 192.168.206.132@tcp -m GET -e local_interrupt -r 2 [ 1520.723458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.32@tcp -d 192.168.206.132@tcp1 -m GET -e local_interrupt -r 2 [ 1520.758183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.32@tcp -d 192.168.206.32@tcp -m GET -e local_interrupt -r 2 [ 1520.795637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.32@tcp -d 192.168.206.32@tcp1 -m GET -e local_interrupt -r 2 [ 1520.831371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.32@tcp1 -d 192.168.206.132@tcp -m GET -e local_interrupt -r 2 [ 1520.869343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.32@tcp1 -d 192.168.206.132@tcp1 -m GET -e local_interrupt -r 2 [ 1520.904854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.32@tcp1 -d 192.168.206.32@tcp -m GET -e local_interrupt -r 2 [ 1520.940900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.32@tcp1 -d 192.168.206.32@tcp1 -m GET -e local_interrupt -r 2 [ 1520.992456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.32@tcp 192.168.206.132@tcp [ 1521.016204] LNet: There was an unexpected network error while writing to 192.168.206.132: rc = -22 [ 1521.023515] LNet: 1 local NIs in recovery (showing 1): 192.168.206.32@tcp [ 1521.330350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1521.370413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1521.408399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1521.443399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1521.487497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1521.525404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1521.551337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1521.582693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1521.615538] LNet: 88646:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1521.648572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1521.701705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1521.735819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1521.763649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1521.794455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1521.816116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1522.257684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.32@tcp -d 192.168.206.132@tcp -m GET -e local_dropped -r 2 [ 1522.293586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.32@tcp -d 192.168.206.132@tcp1 -m GET -e local_dropped -r 2 [ 1522.322205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.32@tcp -d 192.168.206.32@tcp -m GET -e local_dropped -r 2 [ 1522.356712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.32@tcp -d 192.168.206.32@tcp1 -m GET -e local_dropped -r 2 [ 1522.384780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.32@tcp1 -d 192.168.206.132@tcp -m GET -e local_dropped -r 2 [ 1522.412729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.32@tcp1 -d 192.168.206.132@tcp1 -m GET -e local_dropped -r 2 [ 1522.453720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.32@tcp1 -d 192.168.206.32@tcp -m GET -e local_dropped -r 2 [ 1522.498109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.32@tcp1 -d 192.168.206.32@tcp1 -m GET -e local_dropped -r 2 [ 1522.526493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.32@tcp 192.168.206.132@tcp [ 1522.544105] LNet: There was an unexpected network error while writing to 192.168.206.132: rc = -22 [ 1522.784913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1522.829492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1522.881070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1522.929566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1522.957640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1522.983358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1523.029589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1523.076965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1523.101248] LNet: 88766:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1523.130153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1523.166961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1523.214923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1523.252374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1523.297755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1523.337648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1523.811946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.32@tcp -d 192.168.206.132@tcp -m GET -e local_aborted -r 2 [ 1523.885841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.32@tcp -d 192.168.206.132@tcp1 -m GET -e local_aborted -r 2 [ 1523.946750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.32@tcp -d 192.168.206.32@tcp -m GET -e local_aborted -r 2 [ 1523.981540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.32@tcp -d 192.168.206.32@tcp1 -m GET -e local_aborted -r 2 [ 1524.014819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.32@tcp1 -d 192.168.206.132@tcp -m GET -e local_aborted -r 2 [ 1524.040743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.32@tcp1 -d 192.168.206.132@tcp1 -m GET -e local_aborted -r 2 [ 1524.070717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.32@tcp1 -d 192.168.206.32@tcp -m GET -e local_aborted -r 2 [ 1524.106449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.32@tcp1 -d 192.168.206.32@tcp1 -m GET -e local_aborted -r 2 [ 1524.136164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.32@tcp 192.168.206.132@tcp [ 1524.152378] LNet: There was an unexpected network error while writing to 192.168.206.132: rc = -22 [ 1524.376028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1524.409508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1524.443816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1524.489994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1524.536344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1524.577293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1524.618383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1524.647651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1524.670509] LNet: 88886:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1524.701174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1524.725381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1524.763306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1524.795721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1524.829968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1524.874329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1525.451558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.32@tcp -d 192.168.206.132@tcp -m GET -e local_no_route -r 2 [ 1525.480420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.32@tcp -d 192.168.206.132@tcp1 -m GET -e local_no_route -r 2 [ 1525.514597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.32@tcp -d 192.168.206.32@tcp -m GET -e local_no_route -r 2 [ 1525.548577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.32@tcp -d 192.168.206.32@tcp1 -m GET -e local_no_route -r 2 [ 1525.604863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.32@tcp1 -d 192.168.206.132@tcp -m GET -e local_no_route -r 2 [ 1525.658933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.32@tcp1 -d 192.168.206.132@tcp1 -m GET -e local_no_route -r 2 [ 1525.693991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.32@tcp1 -d 192.168.206.32@tcp -m GET -e local_no_route -r 2 [ 1525.727187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.32@tcp1 -d 192.168.206.32@tcp1 -m GET -e local_no_route -r 2 [ 1525.777879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.32@tcp 192.168.206.132@tcp [ 1526.057961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1526.090861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1526.124506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1526.159855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1526.184946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1526.222211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1526.252617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1526.292425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1526.331734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1526.367393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1526.404706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1526.427617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1526.452310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1526.480255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1527.033396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.32@tcp -d 192.168.206.132@tcp -m GET -e local_timeout -r 2 [ 1527.068116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.32@tcp -d 192.168.206.132@tcp1 -m GET -e local_timeout -r 2 [ 1527.096902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.32@tcp -d 192.168.206.32@tcp -m GET -e local_timeout -r 2 [ 1527.123084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.32@tcp -d 192.168.206.32@tcp1 -m GET -e local_timeout -r 2 [ 1527.152720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.32@tcp1 -d 192.168.206.132@tcp -m GET -e local_timeout -r 2 [ 1527.180518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.32@tcp1 -d 192.168.206.132@tcp1 -m GET -e local_timeout -r 2 [ 1527.207055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.32@tcp1 -d 192.168.206.32@tcp -m GET -e local_timeout -r 2 [ 1527.239522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.32@tcp1 -d 192.168.206.32@tcp1 -m GET -e local_timeout -r 2 [ 1527.267804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.32@tcp 192.168.206.132@tcp [ 1527.282350] LNet: There was an unexpected network error while writing to 192.168.206.132: rc = -22 [ 1527.286271] LNet: Skipped 1 previous similar message [ 1527.568976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1527.597181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1527.623565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1527.644225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1527.670133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1527.692768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1527.721529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1527.751377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1527.764468] LNet: 89127:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1527.768685] LNet: 89127:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1527.782313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1527.817717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1527.851080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1527.876727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1527.909668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1527.952366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1528.427501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.32@tcp -d 192.168.206.132@tcp -m GET -e remote_dropped -r 2 [ 1528.464721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.32@tcp -d 192.168.206.132@tcp1 -m GET -e remote_dropped -r 2 [ 1528.500484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.32@tcp -d 192.168.206.32@tcp -m GET -e remote_dropped -r 2 [ 1528.532387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.32@tcp -d 192.168.206.32@tcp1 -m GET -e remote_dropped -r 2 [ 1528.581439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.32@tcp1 -d 192.168.206.132@tcp -m GET -e remote_dropped -r 2 [ 1528.626533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.32@tcp1 -d 192.168.206.132@tcp1 -m GET -e remote_dropped -r 2 [ 1528.663503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.32@tcp1 -d 192.168.206.32@tcp -m GET -e remote_dropped -r 2 [ 1528.724697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.32@tcp1 -d 192.168.206.32@tcp1 -m GET -e remote_dropped -r 2 [ 1528.784889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.32@tcp 192.168.206.132@tcp [ 1528.804802] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.132@tcp [ 1529.013355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1529.036698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1529.067970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1529.104570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1529.133401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1529.171602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1529.227554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1529.271818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1529.308773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1529.357327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1529.392627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1529.430635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1529.485173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1529.530317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1535.013564] LNet: 89322:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1535.022281] LNetError: 89322:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1535.036469] LNet: Removed LNI 192.168.206.32@tcp [ 1535.046497] LNet: Removed LNI 192.168.206.32@tcp1 [ 1535.457106] Key type .llcrypt unregistered [ 1535.459905] Key type ._llcrypt unregistered [ 1552.208652] Key type ._llcrypt registered [ 1552.211572] Key type .llcrypt registered [ 1553.529402] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 23:17:34 (1788837454) [ 1554.105859] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1554.118560] alg: No test for adler32 (adler32-zlib) [ 1555.057576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1555.148400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1555.188675] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 1556.162416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1556.212993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.32@tcp [ 1556.246509] LNetError: 90299:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.206.32@tcp to recovery [ 1556.357443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.206.32@tcp [ 1556.482838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1562.956157] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 23:17:43 (1788837463) [ 1563.190446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1563.250683] LNet: 90885:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1563.276258] LNetError: 90885:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1563.312175] LNet: Removed LNI 192.168.206.32@tcp [ 1563.349124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1563.411406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1563.434614] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 1564.149847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1564.174590] LNet: Added LNI 192.168.206.32@tcp1 [8/256/0/180] [ 1565.261215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.206.32@tcp [ 1565.280847] LNetError: 91154:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.206.32@tcp to recovery [ 1565.472183] LNet: 1 local NIs in recovery (showing 1): 192.168.206.32@tcp [ 1565.478927] LNet: There was an unexpected network error while writing to 192.168.206.32: rc = -22 [ 1566.596879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.206.32@tcp [ 1575.583317] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 23:17:55 (1788837475) [ 1578.101531] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg632-server [ 1580.208554] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 23:18:00 (1788837480) [ 1582.224518] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg632-server [ 1584.056908] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 23:18:04 (1788837484) [ 1586.532174] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg632-server [ 1588.216912] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 23:18:08 (1788837488) [ 1588.435788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1588.464208] LNet: 92777:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1588.471551] LNetError: 92777:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1588.487793] LNet: Removed LNI 192.168.206.32@tcp [ 1588.524072] LNet: Removed LNI 192.168.206.32@tcp1 [ 1588.579090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1588.639065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1588.674761] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 1589.297627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1589.346078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1589.394909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1589.562550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1589.603872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1589.720606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1596.042246] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 23:18:16 (1788837496) [ 1596.222630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1596.261890] LNet: 93508:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1596.265505] LNetError: 93508:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1596.273153] LNetError: 93508:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1596.299962] LNet: Removed LNI 192.168.206.32@tcp [ 1596.311360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1596.370454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1596.408974] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 1603.159794] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 23:18:23 (1788837503) [ 1603.331659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1603.371626] LNet: 94212:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1603.377080] LNetError: 94212:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1603.398162] LNet: Removed LNI 192.168.206.32@tcp [ 1603.420589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1603.453257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1603.481111] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 1611.651557] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 23:18:32 (1788837512) [ 1613.161066] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1614.836567] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 23:18:35 (1788837515) [ 1615.288851] LNet: 95247:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1615.294485] LNetError: 95247:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1615.318149] LNet: Removed LNI 192.168.206.32@tcp [ 1616.060672] Key type .llcrypt unregistered [ 1616.066192] Key type ._llcrypt unregistered [ 1623.429333] Key type ._llcrypt registered [ 1623.432729] Key type .llcrypt registered [ 1623.717846] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1623.753939] alg: No test for adler32 (adler32-zlib) [ 1624.729463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1624.793598] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 1632.264591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.132@tcp [ 1637.577867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.206.132@tcp [ 1637.889983] LNet: 95661:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1637.894684] LNetError: 95661:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1637.905732] LNet: Removed LNI 192.168.206.32@tcp [ 1638.435684] Key type .llcrypt unregistered [ 1638.444808] Key type ._llcrypt unregistered [ 1649.781728] Key type ._llcrypt registered [ 1649.785245] Key type .llcrypt registered [ 1651.133698] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1652.875321] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1654.377522] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 23:19:15 (1788837555) [ 1655.573295] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1657.004892] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 23:19:18 (1788837558) [ 1658.484627] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg632-server [ 1660.058561] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 23:19:21 (1788837561) [ 1661.692150] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg632-server [ 1663.206426] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 23:19:24 (1788837564) [ 1664.340204] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1664.391353] alg: No test for adler32 (adler32-zlib) [ 1665.520353] Lustre: Lustre: Build Version: 2.17.57_107_gb4eb68b [ 1665.744319] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 1667.455385] Key type lgssc registered [ 1668.479510] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1686.267423] Key type lgssc unregistered [ 1686.540758] LNet: 98833:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1686.547744] LNetError: 98833:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1686.573231] LNet: Removed LNI 192.168.206.32@tcp [ 1687.331727] Key type .llcrypt unregistered [ 1687.334596] Key type ._llcrypt unregistered [ 1700.095514] Key type ._llcrypt registered [ 1700.098437] Key type .llcrypt registered [ 1701.202841] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 23:20:02 (1788837602) [ 1701.580282] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1701.594869] alg: No test for adler32 (adler32-zlib) [ 1702.550647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1702.608705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1702.647770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1702.703117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1702.761696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1707.938886] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 23:20:08 (1788837608) [ 1708.015567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1708.031368] LNet: 100239:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1708.054311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1708.150626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1708.172961] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 1708.886113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1708.909732] LNet: 100389:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1708.917186] LNetError: 100389:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1708.935026] LNet: Removed LNI 192.168.206.32@tcp [ 1708.952956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1709.003695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1709.024836] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 1709.442159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1709.466047] LNetError: 100418:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1709.476037] LNet: Removed LNI 192.168.206.32@tcp [ 1709.508649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1709.574567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1709.600921] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 1715.056594] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 23:20:16 (1788837616) [ 1715.224362] LNet: 101007:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1715.233607] LNet: 101007:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1715.247663] LNetError: 101007:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1715.264025] LNet: Removed LNI 192.168.206.32@tcp [ 1715.825957] Key type .llcrypt unregistered [ 1715.830615] Key type ._llcrypt unregistered [ 1721.512987] Key type ._llcrypt registered [ 1721.515639] Key type .llcrypt registered [ 1721.664398] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1721.675319] alg: No test for adler32 (adler32-zlib) [ 1722.635651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1722.720169] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 1722.724412] LNet: *** cfs_fail_loc=e003, val=100*** [ 1723.488771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.32@tcp [ 1723.511426] LNet: *** cfs_fail_loc=e003, val=100*** [ 1723.514824] LNet: Skipped 2 previous similar messages [ 1723.529781] Ping reply received with mapping: [ 1723.534042] found metadata reply NI[192.168.206.32@tcp] [ 1723.541061] 20 ce a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1723.547992] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1723.555702] 5a 5a 5a 5a 5a 5a 5a 5a [ 1723.688646] LNet: 101402:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1723.700192] LNetError: 101402:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1723.742461] LNet: Removed LNI 192.168.206.32@tcp [ 1724.328133] Key type .llcrypt unregistered [ 1724.333577] Key type ._llcrypt unregistered [ 1733.928464] Key type ._llcrypt registered [ 1733.931907] Key type .llcrypt registered [ 1735.087558] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 23:20:35 (1788837635) [ 1735.503251] Key type .llcrypt unregistered [ 1735.507895] Key type ._llcrypt unregistered [ 1740.925172] Key type ._llcrypt registered [ 1740.927437] Key type .llcrypt registered [ 1741.075214] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1741.097834] alg: No test for adler32 (adler32-zlib) [ 1742.219591] LNetError: 102453:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1742.687447] Key type .llcrypt unregistered [ 1742.693250] Key type ._llcrypt unregistered [ 1747.519656] Key type ._llcrypt registered [ 1747.522842] Key type .llcrypt registered [ 1748.873825] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 23:20:49 (1788837649) [ 1749.349919] Key type .llcrypt unregistered [ 1749.355933] Key type ._llcrypt unregistered [ 1755.522724] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1755.538419] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1758.093974] Key type ._llcrypt registered [ 1758.095851] Key type .llcrypt registered [ 1758.253901] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1758.271802] alg: No test for adler32 (adler32-zlib) [ 1759.250111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1759.306152] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1768.163409] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 23:21:09 (1788837669) [ 1768.233240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1768.257113] LNet: 104221:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1768.261141] LNetError: 104221:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1768.284278] LNet: Removed LNI 10.1.2.3@tcp [ 1768.337814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1769.139531] Key type .llcrypt unregistered [ 1769.140981] Key type ._llcrypt unregistered [ 1775.663586] Key type ._llcrypt registered [ 1775.676598] Key type .llcrypt registered [ 1775.982121] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1776.017173] alg: No test for adler32 (adler32-zlib) [ 1777.028992] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 1777.110056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.32@tcp [ 1777.371174] LNet: 104511:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1777.384230] LNetError: 104511:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1777.418654] LNet: Removed LNI 192.168.206.32@tcp [ 1778.176346] Key type .llcrypt unregistered [ 1778.178090] Key type ._llcrypt unregistered [ 1789.222046] Key type ._llcrypt registered [ 1789.223821] Key type .llcrypt registered [ 1790.513334] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 23:21:31 (1788837691) [ 1791.232252] Key type .llcrypt unregistered [ 1791.234466] Key type ._llcrypt unregistered [ 1797.763560] Key type ._llcrypt registered [ 1797.766244] Key type .llcrypt registered [ 1798.056070] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1798.088377] alg: No test for adler32 (adler32-zlib) [ 1804.333973] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 23:21:45 (1788837705) [ 1804.518817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1804.552209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1804.599395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1804.621082] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 1805.369318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1805.416988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1805.460525] LNet: Removed LNI 192.168.206.32@tcp [ 1805.753676] LNet: 106249:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1806.183513] Key type .llcrypt unregistered [ 1806.197325] Key type ._llcrypt unregistered [ 1817.965882] Key type ._llcrypt registered [ 1817.968237] Key type .llcrypt registered [ 1819.290935] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 23:21:59 (1788837719) [ 1819.566302] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1819.582797] alg: No test for adler32 (adler32-zlib) [ 1820.561847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1820.627851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1820.657282] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 1821.369373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.32@tcp [ 1821.407854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.206.32@tcp [ 1821.427507] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1821.431218] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1821.437430] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1821, 0, 0 [ 1821.446609] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1821.449754] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1821.453705] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1821.458062] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1821, 0, 0 [ 1821.465903] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1821.469131] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1821.475373] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1821.478909] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1821, 0, 1871 [ 1821.488071] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1821.490773] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1821.497301] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1821.500190] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1821, 0, 1871 [ 1821.504805] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1827.837189] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 23:22:08 (1788837728) [ 1828.230546] LNet: 107786:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1828.242360] LNetError: 107786:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1829.294294] LNet: Removed LNI 192.168.206.32@tcp [ 1830.021372] Key type .llcrypt unregistered [ 1830.025532] Key type ._llcrypt unregistered [ 1837.211251] Key type ._llcrypt registered [ 1837.212954] Key type .llcrypt registered [ 1837.393898] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1837.406560] alg: No test for adler32 (adler32-zlib) [ 1838.363845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1838.391458] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 1846.326539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.132@tcp [ 1855.571595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1855.615592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1855.644471] LNet: Added LNI 192.168.206.32@tcp1 [8/256/0/180] [ 1856.644609] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1856.742811] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1856.751258] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1856.758459] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1859.212455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1859.247467] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1870.175673] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 23:22:51 (1788837771) [ 1870.347407] LNet: 109293:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1870.360251] LNetError: 109293:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1870.368842] LNet: Removed LNI 192.168.206.32@tcp [ 1871.396702] LNet: Removed LNI 192.168.206.32@tcp1 [ 1871.398324] LNetError: 109293:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1871.401316] LNetError: 109293:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1871.420941] LNet: Removed LNI 10.1.2.3@tcp99 [ 1872.189034] Key type .llcrypt unregistered [ 1872.196926] Key type ._llcrypt unregistered [ 1879.042382] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1879.048218] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1881.785778] Key type ._llcrypt registered [ 1881.788219] Key type .llcrypt registered [ 1882.092706] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1882.113786] alg: No test for adler32 (adler32-zlib) [ 1883.076894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1883.137876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1883.156389] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 1884.026534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1884.052350] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1886.148175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 1886.189985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1886.251230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 1886.270849] LNetError: 110034:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1886.292803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1886.328278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.32@tcp [ 1886.867684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1886.975897] LNet: 110054:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1886.981628] LNetError: 110054:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1886.987306] LNet: Removed LNI 10.1.2.3@tcp [ 1888.038250] LNet: Removed LNI 192.168.206.32@tcp [ 1888.753546] Key type .llcrypt unregistered [ 1888.755705] Key type ._llcrypt unregistered [ 1901.170939] Key type ._llcrypt registered [ 1901.172944] Key type .llcrypt registered [ 1902.567724] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 23:23:23 (1788837803) [ 1902.964467] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1902.982228] alg: No test for adler32 (adler32-zlib) [ 1903.931617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1904.016235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1904.041601] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 1904.790842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.32@tcp [ 1904.926537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg632-client@tcp [ 1911.244309] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 23:23:32 (1788837812) [ 1911.443455] LNet: 111581:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1911.463830] LNetError: 111581:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1911.475267] LNet: Removed LNI 192.168.206.32@tcp [ 1912.110478] Key type .llcrypt unregistered [ 1912.114187] Key type ._llcrypt unregistered [ 1918.419509] Key type ._llcrypt registered [ 1918.421684] Key type .llcrypt registered [ 1918.634277] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1918.678400] alg: No test for adler32 (adler32-zlib) [ 1919.590455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 1919.618689] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 1919.649846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1925.884523] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 23:23:46 (1788837826) [ 1927.092351] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 1928.644524] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 23:23:49 (1788837829) [ 1930.373958] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 1931.990314] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 23:23:52 (1788837832) [ 1932.068337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1932.092756] LNet: 112996:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1932.097936] LNetError: 112996:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1932.112688] LNet: Removed LNI 192.168.206.32@tcp [ 1932.123787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1932.188277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1932.213357] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 1932.804461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1932.846802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1939.329865] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 23:24:00 (1788837840) [ 1939.498617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1939.544059] LNet: 113709:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1939.554707] LNetError: 113709:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1939.571447] LNet: Removed LNI 192.168.206.32@tcp [ 1939.629346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1939.727157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 1939.792482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1940.468464] Key type .llcrypt unregistered [ 1940.473130] Key type ._llcrypt unregistered [ 1952.366981] Key type ._llcrypt registered [ 1952.368365] Key type .llcrypt registered [ 1953.762946] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 23:24:14 (1788837854) [ 1954.162441] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1954.181454] alg: No test for adler32 (adler32-zlib) [ 1955.132994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1955.204982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1955.226219] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 1955.780674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1955.842866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.32@tcp [ 1962.484748] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 23:24:23 (1788837863) [ 1962.791698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1962.822570] LNet: 115264:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1962.833437] LNetError: 115264:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1963.880056] LNet: Removed LNI 192.168.206.32@tcp [ 1963.913030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1963.988411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 1964.042920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 1970.147669] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 23:24:31 (1788837871) [ 1970.321390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1970.337239] LNet: 115841:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1970.370261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1970.508133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1971.051055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1971.087538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1971.164454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 1971.570692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1971.602489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1971.718220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1971.754367] LNet: 115877:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1971.777431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1971.863633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1972.345788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1972.387244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1972.461984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 1972.941954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1972.976731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1979.901946] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 23:24:40 (1788837880) [ 1980.087315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1980.131898] LNet: 116471:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1980.176852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1980.250950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1980.274253] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 1980.949415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1980.982733] LNet: Added LNI 192.168.206.32@tcp2 [8/256/0/180] [ 1981.714308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1981.786715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.32@tcp [ 1982.039304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1982.060860] LNetError: 116757:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1982.068141] LNet: Removed LNI 192.168.206.32@tcp [ 1982.092626] LNet: Removed LNI 192.168.206.32@tcp2 [ 1982.118290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1982.222532] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 1982.237726] LNet: Added LNI 192.168.206.32@tcp2 [8/256/0/180] [ 1982.245110] LNet: 116763:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1982.719206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.32@tcp [ 1982.984678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1984.031704] LNet: Removed LNI 192.168.206.32@tcp [ 1984.038958] LNetError: 116800:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1984.049752] LNetError: 116800:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1984.076051] LNet: Removed LNI 192.168.206.32@tcp2 [ 1984.111507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1984.189801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1984.216809] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 1985.018685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1985.044191] LNet: Added LNI 192.168.206.32@tcp2 [8/256/0/180] [ 1986.056809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.32@tcp [ 1986.128198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1986.231189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.32@tcp [ 1986.451333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1986.476987] LNet: 117085:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1986.483198] LNet: 117085:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1986.488207] LNetError: 117085:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1986.494966] LNet: Removed LNI 192.168.206.32@tcp [ 1987.557824] LNet: Removed LNI 192.168.206.32@tcp2 [ 1987.603523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1987.708918] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 1987.714902] LNet: Added LNI 192.168.206.32@tcp2 [8/256/0/180] [ 1987.721381] LNet: 117091:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1988.379536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.32@tcp [ 1988.651809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1988.687357] LNet: Removed LNI 192.168.206.32@tcp [ 1989.749303] LNet: Removed LNI 192.168.206.32@tcp2 [ 1989.791908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1989.851517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1989.884639] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 1990.632665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1990.654808] LNet: Added LNI 192.168.206.32@tcp2 [8/256/0/180] [ 1991.634474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1991.727566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.32@tcp [ 1992.005537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1992.035835] LNetError: 117407:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1992.048404] LNetError: 117407:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1992.060512] LNet: Removed LNI 192.168.206.32@tcp [ 1992.074240] LNet: Removed LNI 192.168.206.32@tcp2 [ 1992.106872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1992.185863] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 1992.193581] LNet: Added LNI 192.168.206.32@tcp2 [8/256/0/180] [ 1992.199681] LNet: 117413:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1992.844898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.32@tcp [ 1993.012484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1994.079675] LNet: Removed LNI 192.168.206.32@tcp [ 1994.090530] LNet: Removed LNI 192.168.206.32@tcp2 [ 1994.151151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1994.239333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1994.271735] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 1995.033647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1995.054958] LNet: Added LNI 192.168.206.32@tcp2 [8/256/0/180] [ 1995.887816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.32@tcp [ 1995.929585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1996.021237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.32@tcp [ 1996.217430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1996.251768] LNet: 117735:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1996.257497] LNet: 117735:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1997.279215] LNet: Removed LNI 192.168.206.32@tcp [ 1998.310538] LNet: Removed LNI 192.168.206.32@tcp2 [ 1998.379792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1998.499874] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 1998.508627] LNet: Added LNI 192.168.206.32@tcp2 [8/256/0/180] [ 1998.513857] LNet: 117741:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1999.127486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.32@tcp [ 1999.315561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1999.345826] LNet: Removed LNI 192.168.206.32@tcp [ 2000.421287] LNet: Removed LNI 192.168.206.32@tcp2 [ 2000.478732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2000.562192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2000.613373] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 2001.518713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2001.540996] LNet: Added LNI 192.168.206.32@tcp2 [8/256/0/180] [ 2002.250104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2002.287450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2002.375658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.32@tcp [ 2002.683454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2002.706252] LNetError: 118075:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2002.710647] LNetError: 118075:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2002.719219] LNet: Removed LNI 192.168.206.32@tcp [ 2002.742265] LNet: Removed LNI 192.168.206.32@tcp2 [ 2002.793591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2002.855224] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 2002.862957] LNet: Added LNI 192.168.206.32@tcp2 [8/256/0/180] [ 2002.870115] LNet: 118082:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2003.658637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.32@tcp [ 2004.086110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2005.154568] LNet: Removed LNI 192.168.206.32@tcp [ 2005.185962] LNet: Removed LNI 192.168.206.32@tcp2 [ 2005.256174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2005.409572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2005.471467] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 2006.267645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2006.306909] LNet: Added LNI 192.168.206.32@tcp2 [8/256/0/180] [ 2007.312901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.32@tcp [ 2007.359525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2007.401101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2007.480862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.32@tcp [ 2007.950319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2008.991301] LNet: Removed LNI 192.168.206.32@tcp [ 2010.032291] LNet: Removed LNI 192.168.206.32@tcp2 [ 2010.049800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2010.124736] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 2010.129773] LNet: Added LNI 192.168.206.32@tcp2 [8/256/0/180] [ 2010.870137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.32@tcp [ 2018.328606] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2020.026367] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 23:25:20 (1788837920) [ 2020.103958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2020.130849] LNet: 119194:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2020.133426] LNet: 119194:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2020.155556] LNetError: 119194:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2020.166154] LNetError: 119194:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2020.179275] LNet: Removed LNI 192.168.206.32@tcp [ 2021.225960] LNet: Removed LNI 192.168.206.32@tcp2 [ 2021.251280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2021.565666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2021.597129] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 2022.499394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2022.536427] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2023.215578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2023.230955] LNet: Removed LNI 10.1.2.3@tcp [ 2023.243166] LNet: Removed LNI 192.168.206.32@tcp [ 2023.277276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2023.334952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2023.363073] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 2023.367688] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2023.771797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2023.802971] LNet: Removed LNI 192.168.206.32@tcp [ 2023.812495] LNet: Removed LNI 10.1.2.3@tcp [ 2029.620382] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 23:25:30 (1788837930) [ 2029.756362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2029.815716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2029.863775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2029.885485] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 2030.494258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2030.530910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.32@tcp [ 2030.570992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.32@tcp [ 2030.602925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.32@tcp [ 2030.646402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.32@tcp [ 2030.716176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.32@tcp [ 2030.744939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.32@tcp [ 2030.771759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.32@tcp [ 2030.815649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.32@tcp [ 2030.880812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.32@tcp [ 2030.920350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.32@tcp [ 2031.142465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2037.665294] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 23:25:38 (1788837938) [ 2037.755025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2038.825732] LNet: Removed LNI 192.168.206.32@tcp [ 2038.861764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2039.001053] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2039.194036] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2039.202193] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2039.208056] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2041.579509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2041.606825] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 2042.423462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2042.447727] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2044.759191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.32@tcp [ 2044.805810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2044.868648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.206.32@tcp -m PUT -e local_timeout -r 1 [ 2044.947150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2045.024658] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2045.028836] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2045.287780] LNetError: 120876:0:(lib-move.c:4801:lnet_parse()) 192.168.206.32@tcp, src 192.168.206.32@tcp: Dropping PUT (error -108 looking up sender) [ 2046.304277] LNet: Removed LNI 10.1.2.3@tcp [ 2047.338605] LNet: Removed LNI 192.168.206.32@tcp [ 2047.852130] Key type .llcrypt unregistered [ 2047.855535] Key type ._llcrypt unregistered [ 2060.008309] Key type ._llcrypt registered [ 2060.011712] Key type .llcrypt registered [ 2061.141673] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 23:26:02 (1788837962) [ 2061.673091] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2061.691178] alg: No test for adler32 (adler32-zlib) [ 2062.667168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2063.076327] LNet: 122184:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2063.602477] Key type .llcrypt unregistered [ 2063.608934] Key type ._llcrypt unregistered [ 2070.081881] Key type ._llcrypt registered [ 2070.085670] Key type .llcrypt registered [ 2070.306894] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2070.319263] alg: No test for adler32 (adler32-zlib) [ 2071.255633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2071.292406] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 2079.509368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.132@tcp [ 2079.752312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2080.469477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.206.132@tcp [ 2080.548682] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.132@tcp [ 2080.754706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2080.784194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2080.809747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2080.843556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2080.872836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2080.902898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2080.935317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2080.972758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2081.000553] LNet: 122630:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2081.023396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2081.058773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2081.100581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2081.138714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2081.170104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2081.200105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2081.945459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2082.170380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2082.191340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2082.216659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2082.244335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2082.264375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2082.285522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2082.315546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2082.340299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2082.360549] LNet: 122748:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2082.377103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2082.412956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2082.449884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2082.487673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2082.525956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2082.575594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2083.192394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.206.132@tcp [ 2083.515193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2083.550773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2083.604232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2083.654836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2083.688631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2083.743749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2083.781924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2083.825200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2083.851724] LNet: 122865:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2083.872615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2083.902166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2083.948233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2083.990161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2084.020277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2084.055908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2084.778402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2085.083979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2085.135796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2085.182585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2085.232916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2085.289132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2085.334563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2085.371494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2085.409654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2085.434196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2085.464279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2085.496319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2085.525472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2085.570275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2085.616393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2086.017683] LNet: 123028:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2086.028279] LNetError: 123028:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2086.064100] LNet: Removed LNI 192.168.206.32@tcp [ 2086.481914] Key type .llcrypt unregistered [ 2086.486462] Key type ._llcrypt unregistered [ 2103.907556] Key type ._llcrypt registered [ 2103.913866] Key type .llcrypt registered [ 2105.238936] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 23:26:45 (1788838005) [ 2105.722967] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2105.734236] alg: No test for adler32 (adler32-zlib) [ 2106.687786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2107.186450] LNet: 123913:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2107.842502] Key type .llcrypt unregistered [ 2107.848295] Key type ._llcrypt unregistered [ 2114.498556] Key type ._llcrypt registered [ 2114.500476] Key type .llcrypt registered [ 2114.749585] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2114.781673] alg: No test for adler32 (adler32-zlib) [ 2115.759439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2115.811941] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 2122.924348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.132@tcp [ 2131.966583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2131.997524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2132.022866] LNet: Added LNI 192.168.206.32@tcp1 [8/256/0/180] [ 2133.253907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2133.888604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.206.132@tcp [ 2134.151660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2134.205899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2134.257387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2134.304685] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.132@tcp [ 2134.334882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2134.446921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2134.478190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2134.513479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2134.540239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2134.562727] LNet: 124563:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2134.579845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2134.610424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2134.667987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2134.717367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2134.752756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2134.800730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2135.792945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2136.104954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2136.144377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2136.198300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2136.228222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2136.253460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2136.278492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2136.321763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2136.354441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2136.376066] LNet: 124680:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2136.394389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2136.423972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2136.462874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2136.487946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2136.517959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2136.543860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2137.049257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.206.132@tcp [ 2137.333334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2137.373232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2137.427372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2137.461464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2137.492444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2137.513930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2137.538709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2137.561166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2137.578934] LNet: 124797:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2137.598866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2137.633160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2137.652593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2137.675417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2137.704839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2137.743415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2138.354525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2138.643268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2138.671389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2138.698777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2138.736648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2138.780639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2138.834092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2138.890384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2138.934497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2138.968173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2139.004474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2139.070104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2139.112238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2139.147458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2139.196073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2144.472803] LNet: 124996:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2144.479428] LNetError: 124996:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2144.494950] LNet: Removed LNI 192.168.206.32@tcp [ 2144.509741] LNet: Removed LNI 192.168.206.32@tcp1 [ 2145.226839] Key type .llcrypt unregistered [ 2145.228258] Key type ._llcrypt unregistered [ 2160.782145] Key type ._llcrypt registered [ 2160.784946] Key type .llcrypt registered [ 2162.005461] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 23:27:42 (1788838062) [ 2162.470826] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2162.496910] alg: No test for adler32 (adler32-zlib) [ 2163.439416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2163.483763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2163.502799] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 2164.137577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2164.168783] LNet: Added LNI 192.168.206.32@tcp2 [8/256/0/180] [ 2165.247268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.32@tcp [ 2165.352111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2166.432173] LNet: Removed LNI 192.168.206.32@tcp [ 2172.433195] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 23:27:53 (1788838073) [ 2173.744852] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2175.370436] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 23:27:56 (1788838076) [ 2182.416233] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 23:28:03 (1788838083) [ 2189.147958] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 23:28:10 (1788838090) [ 2195.985975] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 23:28:16 (1788838096) [ 2202.291517] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 23:28:23 (1788838103) [ 2209.462219] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 23:28:30 (1788838110) [ 2215.646718] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 23:28:36 (1788838116) [ 2221.695359] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 23:28:42 (1788838122) [ 2228.171034] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 23:28:49 (1788838129) [ 2234.371539] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 23:28:55 (1788838135) [ 2239.873638] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 23:29:00 (1788838140) [ 2245.766631] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 23:29:06 (1788838146) [ 2251.863565] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 23:29:12 (1788838152) [ 2257.781197] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 23:29:18 (1788838158) [ 2263.802646] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 23:29:24 (1788838164) [ 2270.520472] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 23:29:31 (1788838171) [ 2276.557647] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 23:29:37 (1788838177) [ 2282.876742] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 23:29:43 (1788838183) [ 2282.977501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2283.004158] LNet: 136678:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2283.026451] LNetError: 136678:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2283.050973] LNet: Removed LNI 192.168.206.32@tcp2 [ 2283.110162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2283.212573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2283.231564] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 2284.223767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2284.255750] LNet: 136826:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2284.270713] LNetError: 136826:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2284.290072] LNet: Removed LNI 192.168.206.32@tcp [ 2284.325977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2284.384542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2284.410634] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 2290.042603] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 23:29:51 (1788838191) [ 2296.499546] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 23:29:57 (1788838197) [ 2303.259823] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 23:30:04 (1788838204) [ 2310.142894] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 23:30:11 (1788838211) [ 2316.215844] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 23:30:17 (1788838217) [ 2322.884878] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 23:30:23 (1788838223) [ 2329.541968] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 23:30:30 (1788838230) [ 2336.993507] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 23:30:37 (1788838237) [ 2344.330247] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 23:30:45 (1788838245) [ 2350.836955] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 23:30:51 (1788838251) [ 2356.782698] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 23:30:57 (1788838257) [ 2362.675498] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 23:31:03 (1788838263) [ 2369.040818] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2150 sec ========== 23:31:09 (1788838269) [ 2369.422962] LNet: 144290:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2369.434106] LNetError: 144290:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2369.457653] LNet: Removed LNI 192.168.206.32@tcp [ 2370.134569] Key type .llcrypt unregistered [ 2370.136072] Key type ._llcrypt unregistered [ 2381.317711] Key type ._llcrypt registered [ 2381.319885] Key type .llcrypt registered [ 2381.839807] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2381.859062] alg: No test for adler32 (adler32-zlib) [ 2382.917939] Lustre: Lustre: Build Version: 2.17.57_107_gb4eb68b [ 2383.148576] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 2384.847549] Key type lgssc registered [ 2386.026637] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2445.234771] Lustre: Mounted lustre-client - version 2.17.57_107_gb4eb68b [ 2449.905110] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2456.688711] Lustre: Unmounted lustre-client [ 2474.180237] Key type lgssc unregistered [ 2474.397625] LNet: 147237:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2474.404914] LNetError: 147237:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2474.422096] LNet: Removed LNI 192.168.206.32@tcp [ 2475.192200] Key type .llcrypt unregistered [ 2475.193930] Key type ._llcrypt unregistered