[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 456591608 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002389] x2apic enabled [ 0.003021] Switched APIC routing to physical x2apic. [ 0.004015] kvm-guest: setup PV IPIs [ 0.006796] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008012] pid_max: default: 32768 minimum: 301 [ 0.009141] LSM: Security Framework initializing [ 0.010045] Yama: becoming mindful. [ 0.011038] SELinux: Initializing. [ 0.012086] *** VALIDATE selinux *** [ 0.021055] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026478] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027166] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028129] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029112] *** VALIDATE tmpfs *** [ 0.031009] *** VALIDATE proc *** [ 0.032137] *** VALIDATE cgroup *** [ 0.033012] *** VALIDATE cgroup2 *** [ 0.034288] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035141] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037026] Spectre V2 : User space: Vulnerable [ 0.038005] Speculative Store Bypass: Vulnerable [ 0.040795] debug: unmapping init [mem 0xffffffff97459000-0xffffffff97460fff] [ 0.042168] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043695] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044021] ... version: 2 [ 0.045009] ... bit width: 48 [ 0.046009] ... generic registers: 4 [ 0.047013] ... value mask: 0000ffffffffffff [ 0.048015] ... max period: 00007fffffffffff [ 0.049021] ... fixed-purpose events: 3 [ 0.050014] ... event mask: 000000070000000f [ 0.051325] rcu: Hierarchical SRCU implementation. [ 0.053561] smp: Bringing up secondary CPUs ... [ 0.054517] x86: Booting SMP configuration: [ 0.055021] .... node #0, CPUs: #1 #2 #3 [ 0.059470] smp: Brought up 1 node, 4 CPUs [ 0.061022] smpboot: Max logical packages: 1 [ 0.062025] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.227452] node 0 deferred pages initialised in 159ms [ 0.229475] devtmpfs: initialized [ 0.230265] x86/mm: Memory block size: 128MB [ 0.233066] gcov: version magic: 0x41383552 [ 0.235294] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.236099] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.237292] pinctrl core: initialized pinctrl subsystem [ 0.238223] [ 0.238917] ************************************************************* [ 0.239016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.240016] ** ** [ 0.241017] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.242020] ** ** [ 0.243018] ** This means that this kernel is built to expose internal ** [ 0.244016] ** IOMMU data structures, which may compromise security on ** [ 0.245018] ** your system. ** [ 0.246016] ** ** [ 0.247016] ** If you see this message and you are not debugging the ** [ 0.248018] ** kernel, report this immediately to your vendor! ** [ 0.249016] ** ** [ 0.250018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.251015] ************************************************************* [ 0.252733] NET: Registered protocol family 16 [ 0.253511] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.254064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.255073] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.256508] cpuidle: using governor menu [ 0.259830] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.263612] PCI: Using configuration type 1 for base access [ 0.266131] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.277140] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.281049] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.285045] cryptd: max_cpu_qlen set to 1000 [ 0.287245] ACPI: Added _OSI(Module Device) [ 0.289018] ACPI: Added _OSI(Processor Device) [ 0.290020] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.292016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.297894] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.303269] ACPI: Interpreter enabled [ 0.304062] ACPI: PM: (supports S0 S3 S4 S5) [ 0.305015] ACPI: Using IOAPIC for interrupt routing [ 0.307114] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.311485] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.321632] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.323040] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.326020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.328086] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.334486] acpiphp: Slot [2] registered [ 0.336091] acpiphp: Slot [5] registered [ 0.336987] acpiphp: Slot [6] registered [ 0.337118] acpiphp: Slot [3] registered [ 0.338068] acpiphp: Slot [4] registered [ 0.340092] acpiphp: Slot [7] registered [ 0.341120] acpiphp: Slot [8] registered [ 0.342070] acpiphp: Slot [9] registered [ 0.343065] acpiphp: Slot [10] registered [ 0.345095] acpiphp: Slot [11] registered [ 0.346094] acpiphp: Slot [12] registered [ 0.347088] acpiphp: Slot [13] registered [ 0.349168] acpiphp: Slot [14] registered [ 0.350100] acpiphp: Slot [15] registered [ 0.352094] acpiphp: Slot [16] registered [ 0.353106] acpiphp: Slot [17] registered [ 0.355101] acpiphp: Slot [18] registered [ 0.356107] acpiphp: Slot [19] registered [ 0.358083] acpiphp: Slot [20] registered [ 0.359139] acpiphp: Slot [21] registered [ 0.361108] acpiphp: Slot [22] registered [ 0.363191] acpiphp: Slot [23] registered [ 0.365177] acpiphp: Slot [24] registered [ 0.367162] acpiphp: Slot [25] registered [ 0.368215] acpiphp: Slot [26] registered [ 0.370200] acpiphp: Slot [27] registered [ 0.372130] acpiphp: Slot [28] registered [ 0.374281] acpiphp: Slot [29] registered [ 0.376134] acpiphp: Slot [30] registered [ 0.378093] acpiphp: Slot [31] registered [ 0.379124] PCI host bridge to bus 0000:00 [ 0.381034] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.384038] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.387049] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.390045] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.393038] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.395040] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.397317] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.400543] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.403293] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.411610] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.417022] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.420024] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.422030] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.424021] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.428712] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.431939] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.435061] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.438930] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.444020] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.457031] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.462015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.467180] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.476016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.484016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.509017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.518976] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.526020] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.532023] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.551026] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.564695] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.569536] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.571466] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.574478] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.577280] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.583276] iommu: Default domain type: Passthrough [ 0.585612] SCSI subsystem initialized [ 0.587182] ACPI: bus type USB registered [ 0.588150] usbcore: registered new interface driver usbfs [ 0.590093] usbcore: registered new interface driver hub [ 0.593118] usbcore: registered new device driver usb [ 0.596292] pps_core: LinuxPPS API ver. 1 registered [ 0.597010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.600065] PTP clock support registered [ 0.603044] EDAC MC: Ver: 3.0.0 [ 0.604388] PCI: Using ACPI for IRQ routing [ 0.606825] NetLabel: Initializing [ 0.608011] NetLabel: domain hash size = 128 [ 0.609009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.611079] NetLabel: unlabeled traffic allowed by default [ 0.612153] vgaarb: loaded [ 0.614345] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.617026] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.623005] clocksource: Switched to clocksource kvm-clock [ 0.736347] VFS: Disk quotas dquot_6.6.0 [ 0.739163] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.741785] *** VALIDATE ramfs *** [ 0.743172] *** VALIDATE hugetlbfs *** [ 0.744794] pnp: PnP ACPI init [ 0.747533] pnp: PnP ACPI: found 6 devices [ 0.767034] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.770697] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.772974] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.775434] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.778348] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.781101] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.784319] NET: Registered protocol family 2 [ 0.787811] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.792700] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.796472] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.803557] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.806800] TCP: Hash tables configured (established 65536 bind 65536) [ 0.809767] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.813211] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.815965] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.818619] NET: Registered protocol family 1 [ 0.821655] RPC: Registered named UNIX socket transport module. [ 0.824091] RPC: Registered udp transport module. [ 0.825982] RPC: Registered tcp transport module. [ 0.827918] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.830324] NET: Registered protocol family 44 [ 0.832060] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.834282] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.836446] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.839220] PCI: CLS 0 bytes, default 64 [ 0.841084] Unpacking initramfs... [ 2.300574] debug: unmapping init [mem 0xffff9fc2bcc64000-0xffff9fc2bffcffff] [ 2.304897] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.307633] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.310851] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.837042] Initialise system trusted keyrings [ 2.840518] Key type blacklist registered [ 2.842706] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.855097] zbud: loaded [ 2.858713] *** VALIDATE nfs *** [ 2.860156] *** VALIDATE nfs4 *** [ 2.861934] pstore: using deflate compression [ 2.865238] Platform Keyring initialized [ 2.972780] NET: Registered protocol family 38 [ 2.974236] Key type asymmetric registered [ 2.975415] Asymmetric key parser 'x509' registered [ 2.976742] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.979044] io scheduler mq-deadline registered [ 2.980721] io scheduler kyber registered [ 2.981947] io scheduler bfq registered [ 2.983333] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.985675] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.987948] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.990431] ACPI: Power Button [PWRF] [ 2.995531] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.001696] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.015268] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.044461] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.075323] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.091249] Non-volatile memory driver v1.3 [ 3.093040] Linux agpgart interface v0.103 [ 3.131687] virtio_blk virtio1: [vda] 149760 512-byte logical blocks (76.7 MB/73.1 MiB) [ 3.135073] vda: detected capacity change from 0 to 76677120 [ 3.150894] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.153304] vdb: detected capacity change from 0 to 1073741824 [ 3.165813] libphy: Fixed MDIO Bus: probed [ 3.171835] usbcore: registered new interface driver usbserial_generic [ 3.174536] usbserial: USB Serial support registered for generic [ 3.177321] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.182678] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.184900] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.187700] mousedev: PS/2 mouse device common for all mice [ 3.192348] rtc_cmos 00:05: RTC can wake from S4 [ 3.194927] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.202693] rtc_cmos 00:05: registered as rtc0 [ 3.208023] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.210879] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.217355] intel_pstate: CPU model not supported [ 3.223788] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.234443] hid: raw HID events driver (C) Jiri Kosina [ 3.236950] usbcore: registered new interface driver usbhid [ 3.239488] usbhid: USB HID core driver [ 3.241647] drop_monitor: Initializing network drop monitor service [ 3.244771] Initializing XFRM netlink socket [ 3.247322] NET: Registered protocol family 10 [ 3.251102] Segment Routing with IPv6 [ 3.253532] NET: Registered protocol family 17 [ 3.256897] mpls_gso: MPLS GSO support [ 3.262764] RAS: Correctable Errors collector initialized. [ 3.265549] AVX version of gcm_enc/dec engaged. [ 3.267564] AES CTR mode by8 optimization enabled [ 3.350392] sched_clock: Marking stable (3350270904, 0)->(4289579661, -939308757) [ 3.355647] registered taskstats version 1 [ 3.357519] Loading compiled-in X.509 certificates [ 3.359032] zswap: loaded using pool lzo/zbud [ 3.387297] Key type big_key registered [ 3.399849] Key type encrypted registered [ 3.401666] ima: No TPM chip found, activating TPM-bypass! [ 3.403907] ima: Allocated hash algorithm: sha1 [ 3.406175] ima: No architecture policies found [ 3.407890] evm: Initialising EVM extended attributes: [ 3.409426] evm: security.selinux [ 3.410600] evm: security.ima [ 3.411480] evm: security.capability [ 3.412961] evm: HMAC attrs: 0x1 [ 3.415533] rtc_cmos 00:05: setting system clock to 2026-09-04 16:52:00 UTC (1788540720) [ 3.422750] debug: unmapping init [mem 0xffffffff98403000-0xffffffff985fffff] [ 3.426054] debug: unmapping init [mem 0xffffffff97182000-0xffffffff97458fff] [ 3.434471] Write protecting the kernel read-only data: 28672k [ 3.437840] debug: unmapping init [mem 0xffffffff95803000-0xffffffff959fffff] [ 3.440702] debug: unmapping init [mem 0xffffffff96114000-0xffffffff961fffff] [ 3.477074] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.486651] systemd[1]: Detected virtualization kvm. [ 3.488394] systemd[1]: Detected architecture x86-64. [ 3.490536] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.516985] systemd[1]: No hostname configured. [ 3.518846] systemd[1]: Set hostname to . [ 3.521107] random: systemd: uninitialized urandom read (16 bytes read) [ 3.523692] systemd[1]: Initializing machine ID from random generator. [ 3.674880] random: systemd: uninitialized urandom read (16 bytes read) [ 3.678184] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.683701] random: systemd: uninitialized urandom read (16 bytes read) [ 3.686902] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.693183] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... Starting Setup Virtual Console... [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Reached target Swap. [ OK ] Reached target Paths. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [[ 4.307503] random: fast init done  OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.376796] device-mapper: uevent: version 1.0.3 [ 4.379067] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.174996] virtio_net virtio0 ens2: renamed from eth0 [ 5.216223] scsi host0: ata_piix [ 5.263839] scsi host1: ata_piix [ 5.288230] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.291354] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.591268] dracut-initqueue[588]: RTNETLINK answers: File exists [ 9.918424] random: crng init done [ 9.920066] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.411177] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.710973] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.981257] SELinux: Disabled at runtime. [ 12.044340] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.052439] systemd[1]: Detected virtualization kvm. [ 12.054555] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.634596] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.637614] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.642211] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.648369] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.652842] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.660333] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.667855] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Process Core Dump Socket. Mounting Huge Pages File System... Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Control Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-getty.slice. Starting Create list of required st…ce nodes for the current kernel... Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice User and Session Slice. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. Mounting POSIX Message Queue File System... Starting Apply Kernel Variables... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ 12.815415] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Kernel Debug File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. Starting udev Coldplug all Devices... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target Paths. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.224614] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.636780] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.663716] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.762419] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.780248] EDAC sbridge: Ver: 1.1.2 [ 14.921572] Key type dns_resolver registered [ 15.252843] NFS: Registering the id_resolver key type [ 15.255193] Key type id_resolver registered [ 15.257370] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg433-client login: [ 60.704672] libcfs: loading out-of-tree module taints kernel. [ 61.065620] Key type ._llcrypt registered [ 61.132115] Key type .llcrypt registered [ 61.741542] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 61.755887] alg: No test for adler32 (adler32-zlib) [ 63.397862] Lustre: Lustre: Build Version: 2.17.58_2_g4093f3c [ 64.347698] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 66.096335] Key type lgssc registered [ 67.818554] Lustre: Echo OBD driver; http://www.lustre.org/ [ 91.558241] hrtimer: interrupt took 4029487 ns [ 201.026826] Lustre: Mounted lustre-client - version 2.17.58_2_g4093f3c [ 206.381679] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 220.966102] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing check_logdir /tmp/testlogs/ [ 226.573705] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing yml_node [ 226.786824] Lustre: lustre-OST0000-osc-ffff9fc302d14000: disconnect after 23s idle [ 231.341412] Lustre: DEBUG MARKER: Client: 2.17.58.2 [ 233.314494] Lustre: DEBUG MARKER: MDS: 2.17.58.2 [ 236.316883] Lustre: DEBUG MARKER: OSS: 2.17.58.2 [ 238.005325] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Fri Sep 4 12:55:53 EDT 2026 [ 256.014520] Lustre: Unmounted lustre-client [ 282.620823] Key type lgssc unregistered [ 282.935882] LNet: 8409:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 282.951765] LNetError: 8409:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 282.981856] LNet: Removed LNI 192.168.204.33@tcp [ 283.926185] Key type .llcrypt unregistered [ 283.928706] Key type ._llcrypt unregistered [ 307.849274] Key type ._llcrypt registered [ 307.855103] Key type .llcrypt registered [ 308.014990] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 308.026962] alg: No test for adler32 (adler32-zlib) [ 308.892296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 308.943311] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 308.963668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 312.233796] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 312.457548] LNet: 9288:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 312.473872] LNetError: 9288:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 312.492942] LNet: Removed LNI 192.168.204.33@tcp [ 313.029747] Key type .llcrypt unregistered [ 313.032920] Key type ._llcrypt unregistered [ 319.303470] Key type ._llcrypt registered [ 319.308166] Key type .llcrypt registered [ 321.196358] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 12:57:16 (1788541036) [ 321.686265] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 321.715417] alg: No test for adler32 (adler32-zlib) [ 322.679507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 322.850204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 322.900534] LNet: 9696:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 322.942521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 323.033541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 323.075153] LNet: 9703:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 331.415598] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 12:57:26 (1788541046) [ 331.548029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 331.588356] LNet: 10273:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 331.614520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 331.771487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 338.366850] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 12:57:34 (1788541054) [ 338.777410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 338.795351] LNet: 10868:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 338.810798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 338.863829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 345.493577] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 12:57:41 (1788541061) [ 345.624548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 345.685867] LNet: 11447:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 345.733395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 345.854282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 352.068470] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 12:57:47 (1788541067) [ 352.246227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 352.289919] LNet: 12038:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 352.311978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 352.365338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 352.820359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 359.554576] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 12:57:55 (1788541075) [ 359.620345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 359.653752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 359.805685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 360.224885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 360.687994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 366.598230] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 12:58:02 (1788541082) [ 366.829087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 366.844306] LNet: 13225:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 366.850821] LNet: 13225:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 366.873589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 366.953754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 372.983862] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 12:58:08 (1788541088) [ 373.065190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 373.106785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 373.174152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 373.199927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 373.438447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 373.469931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 373.895687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 373.940769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 374.527351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 374.564463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 375.069570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 375.111576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 375.619898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 375.664932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 376.226484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 376.256378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 376.763514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 376.815347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 377.166526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 377.198621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 377.599438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 377.628925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 378.081347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 378.130617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 378.481693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 378.525811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 379.087462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 379.127526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 385.008311] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 12:58:20 (1788541100) [ 385.077853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 385.096412] LNet: 14484:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 385.101962] LNet: 14484:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 385.130236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 385.226329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 385.311516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 391.189702] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 12:58:26 (1788541106) [ 391.507692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 391.582341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 391.702535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 391.753191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 398.924182] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 12:58:34 (1788541114) [ 399.114513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 399.145367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 399.214659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 399.257966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 406.146734] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 12:58:41 (1788541121) [ 406.334649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 406.405853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 406.506183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 406.566294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 412.932859] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 12:58:48 (1788541128) [ 413.034315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 413.102034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 413.183730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 413.215718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 419.350586] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 12:58:55 (1788541135) [ 419.481919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 419.516699] LNet: 17421:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 419.526825] LNet: 17421:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 419.568157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 419.618280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 419.648752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 425.091173] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 12:59:01 (1788541141) [ 425.321289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 425.361904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 425.454800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 425.866234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 432.291140] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 12:59:07 (1788541147) [ 432.409381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 432.481495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 432.721578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 433.127594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 439.665926] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 12:59:15 (1788541155) [ 439.926786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 439.965502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 440.097072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 440.327732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 446.100701] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 12:59:22 (1788541162) [ 446.309572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 446.353606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 446.700800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 446.914465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 452.932665] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 12:59:28 (1788541168) [ 453.064707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 453.122157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 453.199605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 453.251717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 458.818897] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 12:59:34 (1788541174) [ 458.867586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 458.899047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 458.975793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 459.020897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 464.562627] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 12:59:40 (1788541180) [ 464.745809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 464.787386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 464.877843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 470.972519] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 12:59:46 (1788541186) [ 471.115202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 471.171852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 471.328758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 477.565335] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 12:59:53 (1788541193) [ 477.837135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 477.878811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 477.962925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 478.221989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 485.649829] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 13:00:01 (1788541201) [ 485.803036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 485.844546] LNet: 23443:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 485.856844] LNet: 23443:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 485.939966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 486.098684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 486.155580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 493.804973] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 13:00:08 (1788541208) [ 493.874691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 493.944546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 494.108489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 494.155747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 501.941221] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 13:00:17 (1788541217) [ 502.351798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 502.456956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 502.586712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 502.664166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 510.170619] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 13:00:25 (1788541225) [ 510.270056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 510.324222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 510.441386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 510.476543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 516.946306] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 13:00:32 (1788541232) [ 517.162469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 517.217473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 517.282892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 517.363353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 517.386811] LNetError: 25807:0:(peer.c:2366:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 517.448372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 517.472337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 524.314182] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 13:00:39 (1788541239) [ 524.443706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 524.517241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 524.653692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.b0a1Plpeky [ 524.780674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.b0a1Plpeky [ 524.848161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.b0a1Plpeky [ 531.070908] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 13:00:46 (1788541246) [ 531.346610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 531.388700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 531.428657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 531.488990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 531.551096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 537.738784] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 13:00:53 (1788541253) [ 537.919527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 537.991261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 538.050694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 538.104934] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 539.505438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.33@tcp [ 540.870584] Lustre: Lustre: Build Version: 2.17.58_2_g4093f3c [ 542.839299] Key type lgssc registered [ 544.421151] Lustre: Echo OBD driver; http://www.lustre.org/ [ 548.335557] LNetError: 27568:0:(lib-move.c:2151:lnet_handle_find_routed_path()) peer 192.168.204.33@tcp99 has no available nets [ 551.247838] Key type lgssc unregistered [ 551.632661] LNetError: 28686:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 551.664041] LNet: Removed LNI 192.168.204.33@tcp [ 552.350683] Key type .llcrypt unregistered [ 552.353141] Key type ._llcrypt unregistered [ 558.713877] Key type ._llcrypt registered [ 558.715477] Key type .llcrypt registered [ 559.966852] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 13:01:15 (1788541275) [ 560.667401] Key type .llcrypt unregistered [ 560.671907] Key type ._llcrypt unregistered [ 567.496974] Key type ._llcrypt registered [ 567.500183] Key type .llcrypt registered [ 567.728526] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 567.742494] alg: No test for adler32 (adler32-zlib) [ 568.769200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 568.837296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 568.849965] LNet: 29594:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 568.858138] LNet: 29594:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 568.863170] LNet: 29594:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 568.874400] LNet: Message forwarding will be enabled in 2s [ 571.935144] LNet: Message forwarding enabled [ 572.407353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 572.436970] LNet: Message forwarding will stop in 2s to 6s [ 576.034333] LNet: Message forwarding disabled [ 577.033694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 577.080718] LNet: 29693:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 577.087638] LNet: 29693:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 577.093970] LNet: 29693:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 577.111059] LNet: Message forwarding will be enabled in 2s [ 577.277176] LNet: 29706:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 577.868505] Key type .llcrypt unregistered [ 577.874352] Key type ._llcrypt unregistered [ 590.369454] Key type ._llcrypt registered [ 590.371308] Key type .llcrypt registered [ 591.520595] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 13:01:47 (1788541307) [ 591.883333] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 591.898286] alg: No test for adler32 (adler32-zlib) [ 592.798204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 592.862270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 598.957686] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 13:01:54 (1788541314) [ 599.121858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 599.149772] LNet: 31101:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 599.164629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 599.213383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 599.245484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 599.274778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 599.311673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 599.341916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 599.365773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 599.399503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 599.439412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 599.469243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 599.511211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 599.553359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 599.601496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 599.644563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 599.679425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 599.709721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 599.754615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 599.832811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 599.894637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 599.933481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 599.953019] LNetError: 31148:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 606.438603] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 13:02:02 (1788541322) [ 606.601876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 606.637838] LNet: 31716:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 606.692962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 606.846961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 613.274465] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 13:02:08 (1788541328) [ 615.084613] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg433-server [ 617.040206] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 13:02:12 (1788541332) [ 617.160454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 617.199563] LNet: 32642:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 617.242616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 617.346746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 617.413572] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 618.333270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.204.[34-38]@tcp [ 618.354857] LNet: 32783:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 618.414445] LNetError: 32652:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.36@tcp is being used as a gateway but routing feature is not turned on [ 618.443542] LNetError: 32652:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.36@tcp has gone from up to down [ 625.366995] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 13:02:21 (1788541341) [ 627.110683] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg433-server [ 628.699843] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 13:02:24 (1788541344) [ 628.781280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 628.820831] LNet: 33714:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 628.831949] LNetError: 33714:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 628.863855] LNet: Removed LNI 192.168.204.33@tcp [ 628.919373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 628.994497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 629.041119] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 629.830241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.204.[34-39/2]@tcp [ 629.863986] LNet: 33836:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 629.877922] LNet: 33836:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 629.912060] LNetError: 33723:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.36@tcp is being used as a gateway but routing feature is not turned on [ 629.913650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.204.[34-39/2]@tcp [ 629.927444] LNetError: 33723:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 629.927474] LNetError: 33723:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp103 through 192.168.204.36@tcp has gone from up to down [ 629.927479] LNetError: 33723:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 636.310075] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 13:02:32 (1788541352) [ 636.590074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 636.626500] LNet: 34428:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 636.637358] LNetError: 34428:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 636.673955] LNet: Removed LNI 192.168.204.33@tcp [ 636.723754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 636.872961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 636.946060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 636.980587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 637.068738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 637.124741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 637.168626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 637.255465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 637.348289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 637.399563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 637.486788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 637.570151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 637.612612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 637.699124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 637.820597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 637.885707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 637.980497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 638.055626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 638.091829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 638.148149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 638.234944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 638.262642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 638.335976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 638.411861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 638.458719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 638.512648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 638.595929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 638.632460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 638.692959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 638.783982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 638.829346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 638.885812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 638.951919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 638.989979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 639.042564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 645.044897] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 13:02:40 (1788541360) [ 647.029729] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg433-server [ 648.727600] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 13:02:44 (1788541364) [ 651.117629] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg433-server [ 652.620399] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 13:02:48 (1788541368) [ 652.834180] LNet: 35829:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 652.842161] LNet: 35829:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 653.761250] Key type .llcrypt unregistered [ 653.766061] Key type ._llcrypt unregistered [ 660.171371] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 660.255499] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 660.259973] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 660.263360] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 662.825927] Key type ._llcrypt registered [ 662.839027] Key type .llcrypt registered [ 662.996984] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 663.005436] alg: No test for adler32 (adler32-zlib) [ 663.863747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 663.897445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 663.915587] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 664.496477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 664.523208] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 666.563665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 666.589984] LNet: Removed LNI 10.1.2.3@tcp [ 666.708332] LNet: 36541:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 666.717207] LNetError: 36541:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 666.738960] LNet: Removed LNI 192.168.204.33@tcp [ 667.655626] Key type .llcrypt unregistered [ 667.659304] Key type ._llcrypt unregistered [ 679.044822] Key type ._llcrypt registered [ 679.047577] Key type .llcrypt registered [ 680.464535] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 13:03:15 (1788541395) [ 680.774341] Key type .llcrypt unregistered [ 680.776237] Key type ._llcrypt unregistered [ 687.169934] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 687.295588] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 687.301321] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 687.304913] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 689.964839] Key type ._llcrypt registered [ 689.977284] Key type .llcrypt registered [ 690.238066] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 690.269057] alg: No test for adler32 (adler32-zlib) [ 691.227802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 691.289856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 691.331843] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 692.114717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 692.137856] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 695.039115] LNet: 38032:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 695.045826] LNetError: 38032:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 695.066170] LNet: Removed LNI 10.1.2.3@tcp [ 695.084338] LNet: Removed LNI 192.168.204.33@tcp [ 696.003626] Key type .llcrypt unregistered [ 696.007729] Key type ._llcrypt unregistered [ 706.185324] Key type ._llcrypt registered [ 706.187427] Key type .llcrypt registered [ 707.224426] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 13:03:43 (1788541423) [ 707.615209] Key type .llcrypt unregistered [ 707.617627] Key type ._llcrypt unregistered [ 712.675514] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 712.885662] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 712.907736] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 712.921785] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 715.383369] Key type ._llcrypt registered [ 715.385368] Key type .llcrypt registered [ 715.559992] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 715.576743] alg: No test for adler32 (adler32-zlib) [ 716.488729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 716.549257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 716.588247] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 718.235782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 718.257563] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 720.426400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 720.468639] LNet: Removed LNI 10.1.2.3@tcp [ 720.503370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 720.536166] LNet: Removed LNI 10.1.2.31@tcp [ 720.555253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 720.599411] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 721.297842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 721.335443] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 722.296392] LNet: 39950:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 722.300789] LNetError: 39950:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 722.305678] LNet: Removed LNI 10.1.2.31@tcp [ 722.330956] LNet: Removed LNI 10.1.2.3@tcp [ 722.989508] Key type .llcrypt unregistered [ 722.995097] Key type ._llcrypt unregistered [ 734.279107] Key type ._llcrypt registered [ 734.285581] Key type .llcrypt registered [ 735.538148] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 13:04:11 (1788541451) [ 736.072353] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 736.093331] alg: No test for adler32 (adler32-zlib) [ 736.999275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 737.047126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 737.066979] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 737.771920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 737.788576] LNet: 40900:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 737.794713] LNetError: 40900:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 737.808606] LNet: Removed LNI 192.168.204.33@tcp [ 737.822802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 737.876948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.204.33@tcp [ 737.902556] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 738.683528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.204.33@tcp [ 738.720807] LNet: Removed LNI 192.168.204.33@tcp [ 738.859458] LNet: 41035:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 739.456633] Key type .llcrypt unregistered [ 739.460085] Key type ._llcrypt unregistered [ 751.209425] Key type ._llcrypt registered [ 751.211755] Key type .llcrypt registered [ 752.592418] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 13:04:28 (1788541468) [ 754.621529] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg433-server [ 756.258756] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 13:04:32 (1788541472) [ 756.885948] Key type .llcrypt unregistered [ 756.888334] Key type ._llcrypt unregistered [ 763.514832] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 763.703959] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 763.718903] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 763.729520] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 766.337052] Key type ._llcrypt registered [ 766.341330] Key type .llcrypt registered [ 766.591316] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 766.616799] alg: No test for adler32 (adler32-zlib) [ 767.577225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 767.671221] LNet: Added LNI 192.168.204.33@tcp1 [8/256/0/180] [ 769.013226] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 770.882525] LNet: 42869:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 770.895379] LNetError: 42869:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 770.910621] LNet: Removed LNI 192.168.204.33@tcp1 [ 770.928774] LNet: Removed LNI 10.1.2.3@tcp2 [ 771.626135] Key type .llcrypt unregistered [ 771.629123] Key type ._llcrypt unregistered [ 783.814236] Key type ._llcrypt registered [ 783.815994] Key type .llcrypt registered [ 785.111485] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 13:05:00 (1788541500) [ 785.702948] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 785.719649] alg: No test for adler32 (adler32-zlib) [ 786.649766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 786.721724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 786.745151] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 787.676446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 787.699405] LNet: 43836:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 787.707267] LNetError: 43836:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 787.716826] LNet: Removed LNI 192.168.204.33@tcp [ 787.733525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 787.835616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 787.890942] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 795.797891] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 13:05:10 (1788541510) [ 795.889650] LNet: 44419:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 795.907583] LNetError: 44419:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 795.938887] LNet: Removed LNI 192.168.204.33@tcp [ 796.770469] Key type .llcrypt unregistered [ 796.776230] Key type ._llcrypt unregistered [ 803.125396] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 803.245691] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 803.254726] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 803.259864] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 805.621510] Key type ._llcrypt registered [ 805.623827] Key type .llcrypt registered [ 805.783369] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 805.806818] alg: No test for adler32 (adler32-zlib) [ 806.730360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 806.764096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 806.786805] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 807.512794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 807.538770] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 809.877874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 809.925697] LNet: 45168:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 809.933350] LNetError: 45168:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 809.938709] LNet: Removed LNI 10.1.2.3@tcp [ 809.951736] LNet: Removed LNI 192.168.204.33@tcp [ 809.982166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 810.050551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 810.068430] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 810.072103] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 810.458854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 810.509434] LNet: 45195:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 810.520205] LNetError: 45195:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 810.527601] LNetError: 45195:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 810.543371] LNet: Removed LNI 10.1.2.3@tcp [ 810.570925] LNet: Removed LNI 192.168.204.33@tcp [ 810.589137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 810.646500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 810.676359] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 810.686491] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 811.061637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 811.074413] LNet: Added LNI 192.168.204.33@tcp2 [8/256/0/180] [ 811.944075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 811.966407] LNet: 45358:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 811.976399] LNetError: 45358:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 811.986408] LNetError: 45358:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 811.999510] LNet: Removed LNI 10.1.2.3@tcp [ 812.004948] LNet: Removed LNI 192.168.204.33@tcp [ 812.015929] LNet: Removed LNI 192.168.204.33@tcp2 [ 812.047467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 812.160028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 812.199288] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 812.217118] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 812.231691] LNet: Added LNI 192.168.204.33@tcp2 [8/256/0/180] [ 819.474232] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 13:05:34 (1788541534) [ 819.713119] LNet: 45954:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 819.728809] LNetError: 45954:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 819.740384] LNetError: 45954:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 819.748908] LNet: Removed LNI 10.1.2.3@tcp [ 819.751294] LNet: Removed LNI 192.168.204.33@tcp [ 819.766335] LNet: Removed LNI 192.168.204.33@tcp2 [ 820.246183] Key type .llcrypt unregistered [ 820.252262] Key type ._llcrypt unregistered [ 825.547680] Key type ._llcrypt registered [ 825.549869] Key type .llcrypt registered [ 825.704320] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 825.719575] alg: No test for adler32 (adler32-zlib) [ 826.661967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 826.744864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 826.773131] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 827.866940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 827.885639] LNet: 46347:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 827.890939] LNetError: 46347:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 827.910343] LNet: Removed LNI 192.168.204.33@tcp [ 827.954978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 828.028272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 828.051953] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 828.460202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 828.478278] LNet: 46377:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 828.491448] LNetError: 46377:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 828.519793] LNet: Removed LNI 192.168.204.33@tcp [ 828.546835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 828.618365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 828.650541] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 829.370536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 829.421216] LNet: Removed LNI 192.168.204.33@tcp [ 829.460596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 829.570852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 829.610534] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 830.113511] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 830.221299] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 830.232547] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 830.247714] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 832.746106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 832.764799] LNet: 46486:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 832.776262] LNet: 46486:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 832.787737] LNetError: 46486:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 832.797513] LNetError: 46486:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 832.826154] LNet: Removed LNI 192.168.204.33@tcp [ 832.854624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 832.950941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 832.980912] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 838.913418] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 13:05:54 (1788541554) [ 839.057512] LNet: 47076:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 839.069039] LNetError: 47076:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 839.093081] LNet: Removed LNI 192.168.204.33@tcp [ 839.690815] Key type .llcrypt unregistered [ 839.696454] Key type ._llcrypt unregistered [ 845.958125] Key type ._llcrypt registered [ 845.959763] Key type .llcrypt registered [ 846.107674] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 846.135323] alg: No test for adler32 (adler32-zlib) [ 847.103079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 847.166471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 847.199864] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 848.104913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 848.134395] LNet: 47479:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 848.140461] LNetError: 47479:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 848.154440] LNet: Removed LNI 192.168.204.33@tcp [ 848.173732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 848.250279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 848.305840] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 854.491678] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 13:06:10 (1788541570) [ 854.643749] LNet: 48065:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 854.649266] LNetError: 48065:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 854.667269] LNet: Removed LNI 192.168.204.33@tcp [ 855.246579] Key type .llcrypt unregistered [ 855.248646] Key type ._llcrypt unregistered [ 860.617810] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 860.742480] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 860.749018] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 860.751574] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 862.273932] Key type ._llcrypt registered [ 862.276278] Key type .llcrypt registered [ 862.414763] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 862.429898] alg: No test for adler32 (adler32-zlib) [ 863.301065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 863.347278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 863.367842] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 864.215381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 864.255138] LNet: 48508:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 864.275222] LNetError: 48508:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 864.298678] LNet: Removed LNI 192.168.204.33@tcp [ 864.315924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 864.376561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 864.396595] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 870.650677] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 13:06:26 (1788541586) [ 870.964332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 870.996560] LNet: 49106:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 871.002502] LNetError: 49106:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 871.036403] LNet: Removed LNI 192.168.204.33@tcp [ 871.063390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 871.211543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 878.024144] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 13:06:33 (1788541593) [ 878.245355] LNet: 49689:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 878.954862] Key type .llcrypt unregistered [ 878.960693] Key type ._llcrypt unregistered [ 885.316647] Key type ._llcrypt registered [ 885.319266] Key type .llcrypt registered [ 885.534552] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 885.563215] alg: No test for adler32 (adler32-zlib) [ 886.458297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 886.512415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 886.535488] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 887.326791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 887.346181] LNet: 50083:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 887.352517] LNetError: 50083:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 887.369210] LNet: Removed LNI 192.168.204.33@tcp [ 887.406686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 887.478489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 887.523619] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 894.085704] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 13:06:49 (1788541609) [ 894.191044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 894.219624] LNet: 50672:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 894.230280] LNetError: 50672:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 894.254733] LNet: Removed LNI 192.168.204.33@tcp [ 894.291384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 894.353811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 900.745627] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 13:06:56 (1788541616) [ 902.305897] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 903.820926] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 13:06:59 (1788541619) [ 905.470788] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 907.254768] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 13:07:03 (1788541623) [ 909.282423] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 911.259850] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 13:07:06 (1788541626) [ 912.908547] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 914.934886] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 13:07:10 (1788541630) [ 916.669348] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 918.636345] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 13:07:14 (1788541634) [ 918.867137] LNet: 52730:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 919.482618] Key type .llcrypt unregistered [ 919.485431] Key type ._llcrypt unregistered [ 927.066045] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 927.229740] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 927.266760] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 927.282472] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 930.170137] Key type ._llcrypt registered [ 930.177227] Key type .llcrypt registered [ 930.447584] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 930.477886] alg: No test for adler32 (adler32-zlib) [ 931.504922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 931.609528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 931.645610] LNet: Added LNI 192.168.204.33@tcp [8/512/0/180] [ 931.651771] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 931.653864] LNet: 53049:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 931.748960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 931.773950] LNet: 53068:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 931.779906] LNetError: 53068:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 931.785580] LNet: Removed LNI 10.1.2.3@tcp [ 931.790377] LNetError: 53068:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 931.803220] LNet: Removed LNI 192.168.204.33@tcp [ 931.821773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 931.910704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 931.941520] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 932.012616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 932.044790] LNet: 53092:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 932.050811] LNetError: 53092:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 932.077420] LNet: Removed LNI 192.168.204.33@tcp [ 932.119924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 932.248173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 932.297298] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 932.305892] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 932.313495] LNet: 53100:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 932.451365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 932.494686] LNet: 53118:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 932.511373] LNetError: 53118:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 932.521340] LNet: Removed LNI 10.1.2.3@tcp [ 932.524569] LNetError: 53118:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 932.544654] LNet: Removed LNI 192.168.204.33@tcp [ 932.575786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 932.679786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 932.724029] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 932.734887] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 932.744558] LNet: 53126:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 933.091986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 933.110251] LNet: 53152:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 933.124704] LNetError: 53152:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 933.131754] LNet: Removed LNI 10.1.2.3@tcp [ 933.134529] LNetError: 53152:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 933.146376] LNet: Removed LNI 192.168.204.33@tcp [ 933.177078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 933.272290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 933.297863] LNet: Added LNI 192.168.204.33@tcp [8/256/0/182] [ 933.306386] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 933.460411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 933.490693] LNet: 53180:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 933.498991] LNetError: 53180:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 933.504190] LNet: Removed LNI 10.1.2.3@tcp [ 933.506957] LNetError: 53180:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 933.524905] LNet: Removed LNI 192.168.204.33@tcp [ 933.550145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 933.662803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 933.691254] LNet: Added LNI 192.168.204.33@tcp [8/256/0/182] [ 933.698656] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 933.700759] LNet: 53188:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 933.823820] LNet: Removed LNI 10.1.2.3@tcp [ 933.840986] LNet: Removed LNI 192.168.204.33@tcp [ 934.460483] Key type .llcrypt unregistered [ 934.466902] Key type ._llcrypt unregistered [ 945.046870] Key type ._llcrypt registered [ 945.049725] Key type .llcrypt registered [ 946.435944] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 13:07:42 (1788541662) [ 947.891921] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 949.175672] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 13:07:45 (1788541665) [ 949.649487] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 949.678727] alg: No test for adler32 (adler32-zlib) [ 950.528705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 950.577613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 950.599585] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 951.478631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.204.33 [ 951.510650] LNet: 54448:0:(config.c:1740:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 951.518314] LNet: Removed LNI 192.168.204.33@tcp [ 957.391953] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 13:07:52 (1788541672) [ 957.541616] LNet: 55014:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 957.550348] LNetError: 55014:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 957.579737] LNet: Removed LNI 192.168.204.33@tcp [ 958.120433] Key type .llcrypt unregistered [ 958.122687] Key type ._llcrypt unregistered [ 965.065177] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 965.456729] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 968.653654] Key type ._llcrypt registered [ 968.658256] Key type .llcrypt registered [ 968.984647] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 969.026643] alg: No test for adler32 (adler32-zlib) [ 970.555851] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 978.021950] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 13:08:13 (1788541693) [ 978.138737] LNet: 55919:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 978.151867] LNetError: 55919:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 978.185718] LNet: Removed LNI 10.1.2.3@tcp [ 979.226766] Key type .llcrypt unregistered [ 979.229053] Key type ._llcrypt unregistered [ 985.783241] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 986.093292] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 986.102994] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 986.112178] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 988.874534] Key type ._llcrypt registered [ 988.877161] Key type .llcrypt registered [ 989.050844] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 989.067375] alg: No test for adler32 (adler32-zlib) [ 990.020130] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 996.447346] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 13:08:32 (1788541712) [ 998.128646] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 999.875721] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 13:08:35 (1788541715) [ 1000.014885] LNet: 57106:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1000.025088] LNetError: 57106:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1000.044220] LNet: Removed LNI 10.1.2.3@tcp [ 1000.625195] Key type .llcrypt unregistered [ 1000.629448] Key type ._llcrypt unregistered [ 1006.352165] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1006.657255] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1006.663693] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1006.672571] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1009.492315] Key type ._llcrypt registered [ 1009.496544] Key type .llcrypt registered [ 1009.701487] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1009.713522] alg: No test for adler32 (adler32-zlib) [ 1010.594695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1010.654855] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1019.807481] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 13:08:55 (1788541735) [ 1020.298992] LNet: 58339:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1020.307588] LNetError: 58339:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1020.327147] LNet: Removed LNI 10.1.2.3@tcp [ 1021.052490] Key type .llcrypt unregistered [ 1021.055288] Key type ._llcrypt unregistered [ 1027.729736] Key type ._llcrypt registered [ 1027.732289] Key type .llcrypt registered [ 1027.946742] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1027.964901] alg: No test for adler32 (adler32-zlib) [ 1028.890444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1028.917691] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1037.388793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.133@tcp [ 1038.264941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp -m GET -e local_interrupt -r 1 [ 1038.323896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp -m GET -e local_interrupt -r 1 [ 1038.359166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.133@tcp [ 1038.383200] LNet: There was an unexpected network error while writing to 192.168.204.133: rc = -22 [ 1038.666792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1038.700969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1038.752954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1038.820889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1038.921334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1038.961533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1038.988958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1039.028254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1039.059782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1039.077293] LNet: 58776:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1039.101719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1039.142966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1039.187847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1039.200492] LNet: 1 local NIs in recovery (showing 1): 192.168.204.33@tcp [ 1039.204817] LNet: There was an unexpected network error while writing to 192.168.204.33: rc = -22 [ 1039.241118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1039.280278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1039.323549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1039.360940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1040.008360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp -m GET -e local_dropped -r 1 [ 1040.041680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp -m GET -e local_dropped -r 1 [ 1040.098177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.133@tcp [ 1040.435266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1040.467700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1040.502788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1040.560993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1040.597341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1040.632869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1040.679382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1040.727970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1040.783794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1040.815649] LNet: 58887:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1040.853931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1040.898289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1040.929739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1040.977453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1041.016631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1041.057898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1041.093535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1041.775868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp -m GET -e local_aborted -r 1 [ 1041.823823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp -m GET -e local_aborted -r 1 [ 1041.861899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.133@tcp [ 1041.900170] LNet: There was an unexpected network error while writing to 192.168.204.133: rc = -22 [ 1041.905333] LNet: Skipped 1 previous similar message [ 1042.183266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1042.217841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1042.266072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1042.341194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1042.405687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1042.451145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1042.496101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1042.550734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1042.594143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1042.629164] LNet: 58998:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1042.666212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1042.713421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1042.768658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1042.803796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1042.864948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1042.925108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1043.001617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1043.644476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp -m GET -e local_no_route -r 1 [ 1043.697182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp -m GET -e local_no_route -r 1 [ 1043.753879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.133@tcp [ 1044.074416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1044.101924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1044.136367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1044.165431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1044.193810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1044.227790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1044.263283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1044.299529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1044.330111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1044.367493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1044.425891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1044.488452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1044.541804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1044.590819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1044.625577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1044.661455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1045.319653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp -m GET -e local_timeout -r 1 [ 1045.363531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp -m GET -e local_timeout -r 1 [ 1045.396900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.133@tcp [ 1045.417734] LNet: There was an unexpected network error while writing to 192.168.204.133: rc = -22 [ 1045.423296] LNet: Skipped 1 previous similar message [ 1045.746763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1045.788113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1045.825823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1045.875841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1045.939973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1045.983699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1046.030301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1046.064893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1046.112070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1046.135908] LNet: 59221:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1046.146395] LNet: 59221:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1046.164318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1046.206788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1046.239429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1046.264825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1046.288991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1046.331678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1046.367362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1046.965385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp -m GET -e local_error -r 1 [ 1047.015739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp -m GET -e local_error -r 1 [ 1047.067507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.133@tcp [ 1047.385908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1047.426789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1047.464637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1047.492769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1047.526123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1047.558551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1047.584835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1047.612688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1047.666336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1047.703325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1047.769406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1047.857758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1047.938799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1048.010883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1048.055503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1048.106987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1048.417395] LNet: 59373:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1048.423533] LNetError: 59373:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1048.437190] LNet: Removed LNI 192.168.204.33@tcp [ 1048.856683] Key type .llcrypt unregistered [ 1048.864052] Key type ._llcrypt unregistered [ 1067.887578] Key type ._llcrypt registered [ 1067.893320] Key type .llcrypt registered [ 1069.444848] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 13:09:44 (1788541784) [ 1070.222392] Key type .llcrypt unregistered [ 1070.225198] Key type ._llcrypt unregistered [ 1076.056454] Key type ._llcrypt registered [ 1076.059500] Key type .llcrypt registered [ 1076.365452] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1076.390433] alg: No test for adler32 (adler32-zlib) [ 1077.448968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1077.500798] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1086.311435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.133@tcp [ 1100.271530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1100.371303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1100.426670] LNet: Added LNI 192.168.204.33@tcp1 [8/256/0/180] [ 1102.461695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp -m GET -e local_interrupt -r 1 [ 1102.516926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp1 -m GET -e local_interrupt -r 1 [ 1102.572747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp -m GET -e local_interrupt -r 1 [ 1102.643295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp1 -m GET -e local_interrupt -r 1 [ 1102.693327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp -m GET -e local_interrupt -r 1 [ 1102.742698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp1 -m GET -e local_interrupt -r 1 [ 1102.816887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp -m GET -e local_interrupt -r 1 [ 1102.893842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp1 -m GET -e local_interrupt -r 1 [ 1102.927974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.133@tcp [ 1102.944303] LNet: There was an unexpected network error while writing to 192.168.204.133: rc = -22 [ 1102.948975] LNet: 1 local NIs in recovery (showing 1): 192.168.204.33@tcp [ 1103.180736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1103.232989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1103.279765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1103.316575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1103.399252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1103.471245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1103.533370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1103.581752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1103.622714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1103.641859] LNet: 60838:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1103.664474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1103.705823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1103.752316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1103.803986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1103.865812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1103.889204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1103.911928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1104.517873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp -m GET -e local_dropped -r 1 [ 1104.543228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp1 -m GET -e local_dropped -r 1 [ 1104.572581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp -m GET -e local_dropped -r 1 [ 1104.598800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp1 -m GET -e local_dropped -r 1 [ 1104.656640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp -m GET -e local_dropped -r 1 [ 1104.729782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp1 -m GET -e local_dropped -r 1 [ 1104.792359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp -m GET -e local_dropped -r 1 [ 1104.821177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp1 -m GET -e local_dropped -r 1 [ 1104.859994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.133@tcp [ 1104.885963] LNet: There was an unexpected network error while writing to 192.168.204.133: rc = -22 [ 1104.892349] LNet: Skipped 2 previous similar messages [ 1105.093199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1105.134736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1105.185578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1105.261367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1105.299758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1105.350035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1105.408352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1105.475572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1105.528083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1105.554528] LNet: 60961:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1105.595669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1105.643088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1105.681595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1105.721682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1105.759329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1105.789507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1105.821738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1106.423507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp -m GET -e local_aborted -r 1 [ 1106.490532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp1 -m GET -e local_aborted -r 1 [ 1106.577650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp -m GET -e local_aborted -r 1 [ 1106.647422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp1 -m GET -e local_aborted -r 1 [ 1106.701589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp -m GET -e local_aborted -r 1 [ 1106.756583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp1 -m GET -e local_aborted -r 1 [ 1106.806682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp -m GET -e local_aborted -r 1 [ 1106.846716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp1 -m GET -e local_aborted -r 1 [ 1106.936574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.133@tcp [ 1106.971591] LNet: There was an unexpected network error while writing to 192.168.204.133: rc = -22 [ 1106.979701] LNet: Skipped 2 previous similar messages [ 1107.297889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1107.339480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1107.397514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1107.429825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1107.472873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1107.513672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1107.551257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1107.581540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1107.638731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1107.678387] LNet: 61085:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1107.700817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1107.751550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1107.781977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1107.831346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1107.883654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1107.940393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1108.003310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1108.516845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp -m GET -e local_no_route -r 1 [ 1108.553250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp1 -m GET -e local_no_route -r 1 [ 1108.580451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp -m GET -e local_no_route -r 1 [ 1108.602773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp1 -m GET -e local_no_route -r 1 [ 1108.630418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp -m GET -e local_no_route -r 1 [ 1108.663735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp1 -m GET -e local_no_route -r 1 [ 1108.717770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp -m GET -e local_no_route -r 1 [ 1108.750887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp1 -m GET -e local_no_route -r 1 [ 1108.780214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.133@tcp [ 1108.998846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1109.067515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1109.137964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1109.206406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1109.250395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1109.278960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1109.306940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1109.377532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1109.450801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1109.511767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1109.562270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1109.605879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1109.664936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1109.717692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1109.787665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1109.856873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1110.489045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp -m GET -e local_timeout -r 1 [ 1110.542491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp1 -m GET -e local_timeout -r 1 [ 1110.608283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp -m GET -e local_timeout -r 1 [ 1110.672865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp1 -m GET -e local_timeout -r 1 [ 1110.740262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp -m GET -e local_timeout -r 1 [ 1110.785744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp1 -m GET -e local_timeout -r 1 [ 1110.827469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp -m GET -e local_timeout -r 1 [ 1110.905329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp1 -m GET -e local_timeout -r 1 [ 1110.958445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.133@tcp [ 1110.980567] LNet: There was an unexpected network error while writing to 192.168.204.133: rc = -22 [ 1110.988268] LNet: Skipped 5 previous similar messages [ 1111.213424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1111.245432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1111.269507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1111.294335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1111.318869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1111.369227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1111.407390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1111.443603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1111.490168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1111.524571] LNet: 61331:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1111.532414] LNet: 61331:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1111.560899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1111.612229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1111.659626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1111.722175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1111.773785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1111.812234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1111.882938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1112.478605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp -m GET -e local_error -r 1 [ 1112.528750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp1 -m GET -e local_error -r 1 [ 1112.585466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp -m GET -e local_error -r 1 [ 1112.652865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp1 -m GET -e local_error -r 1 [ 1112.719504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp -m GET -e local_error -r 1 [ 1112.781511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp1 -m GET -e local_error -r 1 [ 1112.833947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp -m GET -e local_error -r 1 [ 1112.886437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp1 -m GET -e local_error -r 1 [ 1112.918874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.133@tcp [ 1113.194808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1113.241542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1113.301755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1113.335670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1113.385616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1113.430622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1113.499842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1113.576716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1113.627614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1113.689075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1113.758196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1113.821314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1113.873653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1113.955421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1114.025335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1114.095131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1120.553367] LNet: 61530:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1120.560134] LNetError: 61530:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1120.570245] LNet: Removed LNI 192.168.204.33@tcp [ 1121.641261] LNet: Removed LNI 192.168.204.33@tcp1 [ 1122.321707] Key type .llcrypt unregistered [ 1122.324623] Key type ._llcrypt unregistered [ 1138.916283] Key type ._llcrypt registered [ 1138.918400] Key type .llcrypt registered [ 1140.461719] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 13:10:55 (1788541855) [ 1141.606379] Key type .llcrypt unregistered [ 1141.609425] Key type ._llcrypt unregistered [ 1147.974077] Key type ._llcrypt registered [ 1147.978756] Key type .llcrypt registered [ 1148.248917] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1148.283357] alg: No test for adler32 (adler32-zlib) [ 1149.207849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1149.241214] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1157.258889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.133@tcp [ 1158.246406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp -m GET -e remote_dropped -r 1 [ 1158.272323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp -m GET -e remote_dropped -r 1 [ 1158.299210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.133@tcp [ 1158.316334] LNet: There was an unexpected network error while writing to 192.168.204.133: rc = -22 [ 1158.524654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1158.581501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1158.618813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1158.645801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1158.673876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1158.707658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1158.731432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1158.770701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1158.829403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1158.847220] LNet: 62776:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1158.888579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1158.934486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1158.990285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1159.055880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1159.120954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1159.167371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1159.216362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1159.783621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp -m GET -e remote_error -r 1 [ 1159.813809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp -m GET -e remote_error -r 1 [ 1159.850620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.133@tcp [ 1159.869381] LNet: There was an unexpected network error while writing to 192.168.204.133: rc = -22 [ 1160.094546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1160.118919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1160.148742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1160.175844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1160.204292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1160.233373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1160.262913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1160.296355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1160.324765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1160.341675] LNet: 62888:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1160.357274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1160.385818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1160.418993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1160.453560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1160.483467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1160.535195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1160.571363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1161.256850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp -m GET -e remote_timeout -r 1 [ 1161.293865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp -m GET -e remote_timeout -r 1 [ 1161.343098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.133@tcp [ 1161.374445] LNet: There was an unexpected network error while writing to 192.168.204.133: rc = -22 [ 1161.669497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1161.725103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1161.780436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1161.819923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1161.863238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1161.910928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1161.956592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1162.016452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1162.079361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1162.107601] LNet: 62999:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1162.133791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1162.174555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1162.204516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1162.247462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1162.293486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1162.328409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1162.378624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1162.791533] LNet: 63040:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1162.803951] LNetError: 63040:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1162.821268] LNet: Removed LNI 192.168.204.33@tcp [ 1163.366528] Key type .llcrypt unregistered [ 1163.370822] Key type ._llcrypt unregistered [ 1182.504941] Key type ._llcrypt registered [ 1182.506870] Key type .llcrypt registered [ 1184.132332] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 13:11:39 (1788541899) [ 1184.992468] Key type .llcrypt unregistered [ 1184.994829] Key type ._llcrypt unregistered [ 1192.405947] Key type ._llcrypt registered [ 1192.408881] Key type .llcrypt registered [ 1192.592824] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1192.607850] alg: No test for adler32 (adler32-zlib) [ 1193.516452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1193.547715] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1200.907517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.133@tcp [ 1208.979995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1209.024373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1209.054232] LNet: Added LNI 192.168.204.33@tcp1 [8/256/0/180] [ 1210.316916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp -m GET -e remote_dropped -r 1 [ 1210.354949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp1 -m GET -e remote_dropped -r 1 [ 1210.393970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp -m GET -e remote_dropped -r 1 [ 1210.440850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp1 -m GET -e remote_dropped -r 1 [ 1210.477793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp -m GET -e remote_dropped -r 1 [ 1210.504335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp1 -m GET -e remote_dropped -r 1 [ 1210.537783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp -m GET -e remote_dropped -r 1 [ 1210.578707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp1 -m GET -e remote_dropped -r 1 [ 1210.621834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.133@tcp [ 1210.640486] LNet: There was an unexpected network error while writing to 192.168.204.133: rc = -22 [ 1210.647381] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.133@tcp [ 1210.815477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1210.845911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1210.871901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1210.917105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1210.945981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1210.980263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1211.004745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1211.036552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1211.070645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1211.099272] LNet: 64507:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1211.117983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1211.155096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1211.199534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1211.240711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1211.299691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1211.363329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1211.397381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1211.439184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1211.923074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp -m GET -e remote_error -r 1 [ 1211.971621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp1 -m GET -e remote_error -r 1 [ 1212.006257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp -m GET -e remote_error -r 1 [ 1212.045917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp1 -m GET -e remote_error -r 1 [ 1212.078462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp -m GET -e remote_error -r 1 [ 1212.115862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp1 -m GET -e remote_error -r 1 [ 1212.163787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp -m GET -e remote_error -r 1 [ 1212.213241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp1 -m GET -e remote_error -r 1 [ 1212.254963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.133@tcp [ 1212.289729] LNet: There was an unexpected network error while writing to 192.168.204.133: rc = -22 [ 1212.299215] LNet: Skipped 2 previous similar messages [ 1212.471923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1212.494945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1212.534589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1212.567790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1212.601809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1212.633532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1212.668194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1212.704832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1212.742145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1212.767905] LNet: 64632:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1212.788869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1212.833632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1212.885402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1212.946504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1212.993973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1213.023509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1213.056624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1213.085359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1213.583889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp -m GET -e remote_timeout -r 1 [ 1213.639981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp1 -m GET -e remote_timeout -r 1 [ 1213.679346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp -m GET -e remote_timeout -r 1 [ 1213.738627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp1 -m GET -e remote_timeout -r 1 [ 1213.787404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp -m GET -e remote_timeout -r 1 [ 1213.833233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp1 -m GET -e remote_timeout -r 1 [ 1213.876791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp -m GET -e remote_timeout -r 1 [ 1213.913598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp1 -m GET -e remote_timeout -r 1 [ 1213.953893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.133@tcp [ 1213.969899] LNet: There was an unexpected network error while writing to 192.168.204.133: rc = -22 [ 1214.180456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1214.219364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1214.248704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1214.281341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1214.323524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1214.354571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1214.394550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1214.444194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1214.498379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1214.526746] LNet: 64757:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1214.558679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1214.599966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1214.660887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1214.721349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1214.776348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1214.812566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1214.853473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1214.882916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1219.776543] LNet: 64835:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1219.789810] LNetError: 64835:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1219.798224] LNet: Removed LNI 192.168.204.33@tcp [ 1219.806366] LNet: Removed LNI 192.168.204.33@tcp1 [ 1220.203249] Key type .llcrypt unregistered [ 1220.206296] Key type ._llcrypt unregistered [ 1235.912894] Key type ._llcrypt registered [ 1235.923957] Key type .llcrypt registered [ 1237.029902] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 13:12:32 (1788541952) [ 1238.324449] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1239.919160] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 13:12:35 (1788541955) [ 1240.666981] Key type .llcrypt unregistered [ 1240.668561] Key type ._llcrypt unregistered [ 1245.503726] Key type ._llcrypt registered [ 1245.505427] Key type .llcrypt registered [ 1245.717596] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1245.729147] alg: No test for adler32 (adler32-zlib) [ 1246.603796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1246.630244] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1254.383612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.133@tcp [ 1255.096961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp -m GET -e network_timeout -r 1 [ 1255.165745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp -m GET -e network_timeout -r 1 [ 1255.205136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.133@tcp [ 1255.222148] LNet: There was an unexpected network error while writing to 192.168.204.133: rc = -22 [ 1255.507291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1255.529191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1255.547242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1255.563075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1255.587504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1255.622778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1255.659496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1255.710439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1255.762791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1255.776362] LNet: 66376:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1255.788975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1255.829141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1255.842294] LNet: 1 local NIs in recovery (showing 1): 192.168.204.33@tcp [ 1255.854355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1255.873669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1255.926483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1255.968765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1256.238501] LNet: 66415:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1256.254099] LNetError: 66415:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1256.275329] LNet: Removed LNI 192.168.204.33@tcp [ 1256.584531] Key type .llcrypt unregistered [ 1256.587068] Key type ._llcrypt unregistered [ 1276.169467] Key type ._llcrypt registered [ 1276.174922] Key type .llcrypt registered [ 1276.501252] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1276.535323] alg: No test for adler32 (adler32-zlib) [ 1277.556520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1277.593865] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1288.139431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.133@tcp [ 1299.046618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1299.114797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1299.160427] LNet: Added LNI 192.168.204.33@tcp1 [8/256/0/180] [ 1300.570307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp -m GET -e network_timeout -r 1 [ 1300.610232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp1 -m GET -e network_timeout -r 1 [ 1300.667400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp -m GET -e network_timeout -r 1 [ 1300.726454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp1 -m GET -e network_timeout -r 1 [ 1300.769660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp -m GET -e network_timeout -r 1 [ 1300.797924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp1 -m GET -e network_timeout -r 1 [ 1300.829691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp -m GET -e network_timeout -r 1 [ 1300.873649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp1 -m GET -e network_timeout -r 1 [ 1300.919256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.133@tcp [ 1300.944424] LNet: There was an unexpected network error while writing to 192.168.204.133: rc = -22 [ 1301.160645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1301.162605] LNet: 1 local NIs in recovery (showing 1): 192.168.204.33@tcp [ 1301.184838] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.133@tcp [ 1301.200761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1301.232089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1301.260797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1301.279904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1301.313546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1301.349247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1301.378982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1301.430833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1301.489751] LNet: 67294:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1301.539640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1301.585485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1301.646151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1301.720931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1301.765837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1301.827906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1306.278166] LNet: There was an unexpected network error while writing to 192.168.204.133: rc = -22 [ 1306.291388] LNet: Skipped 2 previous similar messages [ 1306.641595] LNet: 67368:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1306.653902] LNetError: 67368:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1306.665638] LNet: Removed LNI 192.168.204.33@tcp [ 1306.695175] LNet: Removed LNI 192.168.204.33@tcp1 [ 1306.964743] Key type .llcrypt unregistered [ 1306.968777] Key type ._llcrypt unregistered [ 1322.051429] Key type ._llcrypt registered [ 1322.053651] Key type .llcrypt registered [ 1323.399104] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 13:13:58 (1788542038) [ 1323.988223] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1324.004330] alg: No test for adler32 (adler32-zlib) [ 1324.986182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1325.104798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1325.152716] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1326.113509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1326.139046] LNet: Added LNI 192.168.204.33@tcp1 [8/256/0/180] [ 1327.335107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.33@tcp [ 1327.419400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1327.462875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1327.529107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1327.595926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1327.654379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1327.704788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.33@tcp [ 1327.732327] LNetError: 68475:0:(api-ni.c:4483:lnet_ni_set_healthv()) manually adding local NI 192.168.204.33@tcp to recovery [ 1328.159321] LNet: 1 local NIs in recovery (showing 1): 192.168.204.33@tcp [ 1328.160850] LNet: There was an unexpected network error while writing to 192.168.204.33: rc = -22 [ 1330.216168] LNet: There was an unexpected network error while writing to 192.168.204.33: rc = -22 [ 1331.401706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1334.305837] LNet: There was an unexpected network error while writing to 192.168.204.33: rc = -22 [ 1335.075884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1335.114769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1335.180651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1335.199300] LNet: 68583:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1335.204068] LNetError: 68583:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1335.211951] LNet: Removed LNI 192.168.204.33@tcp [ 1335.222980] LNet: Removed LNI 192.168.204.33@tcp1 [ 1335.242985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1335.307400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1335.329547] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1336.130219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1336.152263] LNet: Added LNI 192.168.204.33@tcp1 [8/256/0/180] [ 1337.125343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.33@tcp [ 1337.162801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1337.239227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1337.319750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1337.372847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1337.433350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1337.484463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1337.531882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.33@tcp [ 1337.569760] LNetError: 68855:0:(api-ni.c:4483:lnet_ni_set_healthv()) manually adding local NI 192.168.204.33@tcp to recovery [ 1338.336554] LNet: There was an unexpected network error while writing to 192.168.204.33: rc = -22 [ 1341.211910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1344.479514] LNet: There was an unexpected network error while writing to 192.168.204.33: rc = -22 [ 1344.485932] LNet: Skipped 1 previous similar message [ 1349.645809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1349.687846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1349.725586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1356.580825] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 13:14:32 (1788542072) [ 1356.714885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1356.753627] LNet: 69565:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1356.764791] LNetError: 69565:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1356.775883] LNetError: 69565:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1356.784657] LNet: Removed LNI 192.168.204.33@tcp [ 1356.793490] LNet: Removed LNI 192.168.204.33@tcp1 [ 1356.834880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1356.889342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1356.909486] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1357.656262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1357.677823] LNet: Added LNI 192.168.204.33@tcp1 [8/256/0/180] [ 1358.505886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.33@tcp [ 1358.638962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1358.680783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1358.733439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1358.780088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1358.845501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1358.886264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.33@tcp --health 0 [ 1358.943290] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.33@tcp [ 1358.990734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1359.967575] LNet: There was an unexpected network error while writing to 192.168.204.33: rc = -22 [ 1359.977586] LNet: Skipped 1 previous similar message [ 1374.529644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1374.692472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1374.735805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1374.762960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.33@tcp --health 500 [ 1374.924764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1380.597789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1380.685314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1380.734634] LNet: 70056:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1380.740324] LNetError: 70056:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1380.750723] LNetError: 70056:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1380.762512] LNet: Removed LNI 192.168.204.33@tcp [ 1380.773919] LNet: Removed LNI 192.168.204.33@tcp1 [ 1380.848677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1380.940804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1380.965610] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1381.862204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1381.880673] LNet: Added LNI 192.168.204.33@tcp1 [8/256/0/180] [ 1382.710514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.33@tcp [ 1382.754822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1382.846377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1382.899854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1382.945450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1382.987961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1383.037527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1383.080860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.33@tcp --health 0 [ 1384.992281] LNet: There was an unexpected network error while writing to 192.168.204.33: rc = -22 [ 1384.998597] LNet: Skipped 3 previous similar messages [ 1385.543334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1394.930406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1394.979606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1395.029859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1402.692276] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 13:15:18 (1788542118) [ 1403.335915] LNet: 71103:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1403.340864] LNetError: 71103:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1403.350878] LNetError: 71103:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1403.358865] LNet: Removed LNI 192.168.204.33@tcp [ 1403.373629] LNet: Removed LNI 192.168.204.33@tcp1 [ 1404.103049] Key type .llcrypt unregistered [ 1404.104869] Key type ._llcrypt unregistered [ 1409.946407] Key type ._llcrypt registered [ 1409.948538] Key type .llcrypt registered [ 1410.186069] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1410.200293] alg: No test for adler32 (adler32-zlib) [ 1411.128373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1411.152423] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1419.927655] Lustre: DEBUG MARKER: Initial discovery [ 1419.995357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.133@tcp [ 1423.465321] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1423.594679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1423.693102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.133@tcp [ 1423.732148] LNet: There was an unexpected network error while writing to 192.168.204.133: rc = -22 [ 1423.781064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1423.850575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.204.133@tcp -r 1 -m GET -l 3 [ 1423.894818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.204.133@tcp -r 1 -m GET -e local_error [ 1423.955656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.133@tcp -r 1 -m PUT -l 6 [ 1424.544250] LNet: 1 local NIs in recovery (showing 1): 192.168.204.33@tcp [ 1425.612791] Lustre: DEBUG MARKER: Force oleg433-client.virtnet to discover 192.168.204.133@tcp (in background) [ 1425.948626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.133@tcp [ 1427.435663] Lustre: DEBUG MARKER: Force oleg433-server to discover 192.168.204.33@tcp [ 1429.027791] LNet: There was an unexpected network error while writing to 192.168.204.133: rc = -22 [ 1430.810995] Lustre: DEBUG MARKER: Wait for 71864 [ 1432.653469] Lustre: DEBUG MARKER: Finished wait on 71864 [ 1432.743492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1432.780536] LNet: There was an unexpected network error while writing to 192.168.204.133: rc = -22 [ 1432.799460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1433.057958] LNet: 72312:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1433.065213] LNetError: 72312:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1434.085812] LNet: Removed LNI 192.168.204.33@tcp [ 1434.817502] Key type .llcrypt unregistered [ 1434.820647] Key type ._llcrypt unregistered [ 1450.086646] Key type ._llcrypt registered [ 1450.088426] Key type .llcrypt registered [ 1451.396250] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 13:16:07 (1788542167) [ 1451.880568] Key type .llcrypt unregistered [ 1451.882441] Key type ._llcrypt unregistered [ 1458.272898] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1458.444765] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1458.448169] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1458.451887] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1461.140975] Key type ._llcrypt registered [ 1461.145071] Key type .llcrypt registered [ 1461.353252] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1461.372166] alg: No test for adler32 (adler32-zlib) [ 1462.278514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1462.328645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1462.350985] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1463.295069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1463.319917] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1472.148388] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 13:16:28 (1788542188) [ 1472.229376] LNet: 74375:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1472.235707] LNetError: 74375:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1472.246795] LNet: Removed LNI 10.1.2.3@tcp [ 1472.259696] LNet: Removed LNI 192.168.204.33@tcp [ 1472.764776] Key type .llcrypt unregistered [ 1472.769807] Key type ._llcrypt unregistered [ 1478.183431] Key type ._llcrypt registered [ 1478.186413] Key type .llcrypt registered [ 1478.416750] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1478.441770] alg: No test for adler32 (adler32-zlib) [ 1479.416243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1479.504533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1479.546573] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1480.477864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1480.505941] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1481.729908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.204.33@tcp [ 1482.783512] LNetError: Unexpected error -22 connecting to 192.168.204.33@tcp at host 192.168.204.33:988 [ 1488.250971] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 13:16:44 (1788542204) [ 1488.395616] LNet: 75505:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1488.425078] LNetError: 75505:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1488.434309] LNet: Removed LNI 10.1.2.3@tcp [ 1488.448081] LNet: Removed LNI 192.168.204.33@tcp [ 1489.151177] Key type .llcrypt unregistered [ 1489.161552] Key type ._llcrypt unregistered [ 1496.314296] Key type ._llcrypt registered [ 1496.318889] Key type .llcrypt registered [ 1496.585493] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1496.607271] alg: No test for adler32 (adler32-zlib) [ 1497.619810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1497.695062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1497.734936] LNet: Added LNI 192.168.204.33@tcp1 [8/256/0/180] [ 1498.664878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1498.701255] LNet: Added LNI 192.168.204.33@tcp2 [8/256/0/180] [ 1499.789930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.204.33@tcp1 --nid 192.168.204.33@tcp2 [ 1504.059935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.33@tcp1 192.168.204.33@tcp2 [ 1504.085535] LNetError: 76346:0:(lib-move.c:2094:lnet_handle_find_routed_path()) Unable to send message from 192.168.204.33@tcp1 to 192.168.204.33@tcp2 - Route table may be misconfigured [ 1504.116582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.33@tcp2 192.168.204.33@tcp1 [ 1510.818597] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 13:17:06 (1788542226) [ 1510.934940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1510.978077] LNet: 76915:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1510.986732] LNetError: 76915:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1512.031483] LNet: Removed LNI 192.168.204.33@tcp1 [ 1512.034762] LNetError: 76915:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1513.065760] LNet: Removed LNI 192.168.204.33@tcp2 [ 1513.127960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1513.219965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1513.271128] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1514.108976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1514.147336] LNet: Added LNI 192.168.204.33@tcp1 [8/256/0/180] [ 1515.109541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.33@tcp [ 1515.148901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.33@tcp 192.168.204.33@tcp [ 1515.185980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.33@tcp1 192.168.204.33@tcp1 [ 1515.253519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.33@tcp -d 192.168.204.33@tcp -e network_timeout [ 1515.282602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.33@tcp -d 192.168.204.33@tcp1 -e network_timeout [ 1515.310391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp -e network_timeout [ 1515.341943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp1 -e network_timeout [ 1515.382577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 1515.400607] LNet: There was an unexpected network error while writing to 192.168.204.33: rc = -22 [ 1515.415094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1515.515730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1515.608713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1516.257079] LNet: 1 local NIs in recovery (showing 1): 192.168.204.33@tcp [ 1522.379665] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 13:17:18 (1788542238) [ 1522.493198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1522.552308] LNet: 77774:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1522.562576] LNetError: 77774:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1522.570095] LNet: Removed LNI 192.168.204.33@tcp [ 1523.622400] LNet: Removed LNI 192.168.204.33@tcp1 [ 1523.716055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1523.925707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1524.209904] LNet: 77819:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1524.644824] Key type .llcrypt unregistered [ 1524.648413] Key type ._llcrypt unregistered [ 1537.365679] Key type ._llcrypt registered [ 1537.369428] Key type .llcrypt registered [ 1538.679276] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 13:17:34 (1788542254) [ 1539.068317] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1539.080389] alg: No test for adler32 (adler32-zlib) [ 1540.085110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1541.393029] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1542.931874] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 13:17:38 (1788542258) [ 1543.157570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1543.188819] LNet: 78915:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1543.213698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1543.282235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1543.309107] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1543.959935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1543.988039] LNet: Added LNI 192.168.204.33@tcp1 [8/256/0/180] [ 1544.915212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 1544.955471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp1 [ 1544.986945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.33@tcp1 [ 1551.804206] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 13:17:47 (1788542267) [ 1554.393871] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg433-server [ 1556.175197] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 13:17:51 (1788542271) [ 1558.223862] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg433-server [ 1560.082925] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 13:17:55 (1788542275) [ 1562.190654] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg433-server [ 1563.826768] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 13:17:59 (1788542279) [ 1565.732868] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg433-server [ 1567.642496] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 13:18:03 (1788542283) [ 1569.537778] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg433-server [ 1571.327587] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 13:18:06 (1788542286) [ 1573.013795] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg433-server [ 1574.786282] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 13:18:10 (1788542290) [ 1576.856496] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg433-server [ 1578.493775] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 13:18:14 (1788542294) [ 1580.267409] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg433-server [ 1581.663362] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 13:18:17 (1788542297) [ 1583.542191] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg433-server [ 1585.383344] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 13:18:21 (1788542301) [ 1585.523540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1585.549221] LNet: 82789:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1585.554979] LNetError: 82789:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1586.591179] LNet: Removed LNI 192.168.204.33@tcp [ 1586.595577] LNetError: 82789:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1586.612600] LNet: Removed LNI 192.168.204.33@tcp1 [ 1586.641854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1586.707555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1586.736684] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1587.499977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1587.581710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 1587.766099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1587.795400] LNet: 82941:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1587.800926] LNetError: 82941:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1587.817260] LNet: Removed LNI 192.168.204.33@tcp [ 1587.839225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1587.887859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1587.911783] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1588.685275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 1589.963987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1589.986269] LNet: 83095:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1589.992617] LNetError: 83095:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1590.010060] LNet: Removed LNI 192.168.204.33@tcp [ 1590.028278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1590.091886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1590.120374] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1590.874447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1590.937476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 1591.165212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1592.233801] LNet: Removed LNI 192.168.204.33@tcp [ 1592.300269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1592.344299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1592.385682] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1593.140930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 1593.377507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1594.410504] LNet: Removed LNI 192.168.204.33@tcp [ 1594.431194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1594.490277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1594.517813] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1595.153963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1595.257852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 1595.451540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1595.482922] LNet: 83535:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1595.493348] LNet: 83535:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1595.500171] LNetError: 83535:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1595.508192] LNetError: 83535:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1595.522762] LNet: Removed LNI 192.168.204.33@tcp [ 1595.554412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1595.629321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1595.651900] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1596.359524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 1596.558887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1597.606560] LNet: Removed LNI 192.168.204.33@tcp [ 1597.653502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1597.751487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1597.815953] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1598.622528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1598.708807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 1598.955687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1600.043781] LNet: Removed LNI 192.168.204.33@tcp [ 1600.068287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1600.150971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1600.176626] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1600.997665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 1601.207830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1602.282229] LNet: Removed LNI 192.168.204.33@tcp [ 1602.297571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1602.369825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1602.413638] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1603.131952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1603.209757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 1603.464691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1603.488660] LNet: 84120:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1603.494025] LNet: 84120:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1603.500683] LNetError: 84120:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1603.505318] LNetError: 84120:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1604.521392] LNet: Removed LNI 192.168.204.33@tcp [ 1604.562782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1604.640365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1604.687625] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1605.453751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 1605.766962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1606.827470] LNet: Removed LNI 192.168.204.33@tcp [ 1606.873607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1606.945416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1606.968746] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1607.643054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1607.749405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 1608.045512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1609.128710] LNet: Removed LNI 192.168.204.33@tcp [ 1609.178417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1609.256765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1609.284883] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1610.144442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 1610.401951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1611.495941] LNet: Removed LNI 192.168.204.33@tcp [ 1611.543776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1611.666850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1611.723756] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1612.573362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1612.632141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1612.661523] LNet: Removed LNI 192.168.204.33@tcp [ 1612.685149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1612.744871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1612.785082] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1613.667872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1613.770039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1613.812663] LNet: Removed LNI 192.168.204.33@tcp [ 1613.851341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1613.953667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1614.060906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1614.126443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1614.214700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1614.309923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1614.344685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1614.406346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1614.462765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1614.490424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1614.558104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1614.597625] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1615.393721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1615.450488] LNet: Removed LNI 192.168.204.33@tcp [ 1615.486365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1615.628831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1623.433058] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 13:18:59 (1788542339) [ 1623.655769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1623.686410] LNet: 85577:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1623.699540] LNet: 85577:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 1623.753766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1623.801379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1623.822801] LNet: Added LNI 192.168.204.33@tcp231 [8/256/0/180] [ 1625.428558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1625.458979] LNetError: 85718:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1625.471082] LNetError: 85718:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1625.489886] LNet: Removed LNI 192.168.204.33@tcp231 [ 1625.514870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1625.578439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1625.604400] LNet: Added LNI 192.168.204.33@tcp231 [8/256/0/180] [ 1625.613337] LNet: 85726:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1626.036320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1626.104266] LNet: Removed LNI 192.168.204.33@tcp231 [ 1626.121510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1626.140565] LNet: Added LNI 192.168.204.33@tcp231 [8/256/0/180] [ 1627.152719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1627.184984] LNet: Removed LNI 192.168.204.33@tcp231 [ 1627.206349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1627.299401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1627.334456] LNet: Added LNI 192.168.204.33@tcp231 [8/256/0/0] [ 1627.341626] LNet: 85878:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1627.695364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1627.718654] LNet: Removed LNI 192.168.204.33@tcp231 [ 1627.741703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1627.786494] LNet: Added LNI 192.168.204.33@tcp231 [8/256/0/0] [ 1628.927876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1628.951831] LNet: Removed LNI 192.168.204.33@tcp231 [ 1628.973273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1629.046643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1629.075386] LNet: Added LNI 192.168.204.33@tcp231 [8/256/0/60] [ 1629.082480] LNet: 86029:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1629.621744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1629.658885] LNet: Removed LNI 192.168.204.33@tcp231 [ 1629.686056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1629.726671] LNet: Added LNI 192.168.204.33@tcp231 [8/256/0/60] [ 1630.838975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1630.862146] LNet: Removed LNI 192.168.204.33@tcp231 [ 1630.882617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1630.935625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1630.958701] LNet: Added LNI 192.168.204.33@tcp231 [8/256/0/180] [ 1631.336461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1631.376716] LNet: Removed LNI 192.168.204.33@tcp231 [ 1631.395395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1631.427277] LNet: Added LNI 192.168.204.33@tcp231 [8/256/0/180] [ 1639.388209] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 13:19:14 (1788542354) [ 1639.712993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1639.774618] LNet: Removed LNI 192.168.204.33@tcp231 [ 1639.827219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1639.911074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1639.936366] LNet: Added LNI 192.168.204.33@tcp232 [8/256/0/180] [ 1640.732900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1640.773103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1640.802968] LNet: Removed LNI 192.168.204.33@tcp232 [ 1640.820500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1640.864215] LNet: Added LNI 192.168.204.33@tcp232 [8/256/0/180] [ 1641.802349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1641.859254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1641.892446] LNet: Removed LNI 192.168.204.33@tcp232 [ 1641.917155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1641.981541] LNet: Added LNI 192.168.204.33@tcp232 [8/256/0/180] [ 1643.256307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1643.341811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1643.384761] LNet: Removed LNI 192.168.204.33@tcp232 [ 1643.404462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1643.443453] LNet: Added LNI 192.168.204.33@tcp232 [8/256/0/180] [ 1644.451135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1644.510685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1644.559845] LNet: Removed LNI 192.168.204.33@tcp232 [ 1644.578630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1651.914972] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 13:19:27 (1788542367) [ 1653.277274] Key type .llcrypt unregistered [ 1653.282657] Key type ._llcrypt unregistered [ 1660.520646] Key type ._llcrypt registered [ 1660.523634] Key type .llcrypt registered [ 1660.919449] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1660.948217] alg: No test for adler32 (adler32-zlib) [ 1661.850764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1661.871052] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1668.833990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.133@tcp [ 1677.076924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1677.114607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1677.137215] LNet: Added LNI 192.168.204.33@tcp1 [8/256/0/180] [ 1678.461460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp -m GET -e local_interrupt -r 2 [ 1678.492464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp1 -m GET -e local_interrupt -r 2 [ 1678.566902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp -m GET -e local_interrupt -r 2 [ 1678.605200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp1 -m GET -e local_interrupt -r 2 [ 1678.676894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp -m GET -e local_interrupt -r 2 [ 1678.744153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp1 -m GET -e local_interrupt -r 2 [ 1678.787728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp -m GET -e local_interrupt -r 2 [ 1678.831539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp1 -m GET -e local_interrupt -r 2 [ 1678.902435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.33@tcp 192.168.204.133@tcp [ 1678.939441] LNet: There was an unexpected network error while writing to 192.168.204.133: rc = -22 [ 1678.948905] LNet: 1 local NIs in recovery (showing 1): 192.168.204.33@tcp [ 1679.215792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1679.239862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1679.271797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1679.318195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1679.348740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1679.399660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1679.441658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1679.488339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1679.525637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1679.560428] LNet: 88680:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1679.577492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1679.610593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1679.636636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1679.677600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1679.720310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1679.756391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1680.296748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp -m GET -e local_dropped -r 2 [ 1680.333260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp1 -m GET -e local_dropped -r 2 [ 1680.369919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp -m GET -e local_dropped -r 2 [ 1680.404320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp1 -m GET -e local_dropped -r 2 [ 1680.442697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp -m GET -e local_dropped -r 2 [ 1680.480951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp1 -m GET -e local_dropped -r 2 [ 1680.552500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp -m GET -e local_dropped -r 2 [ 1680.595748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp1 -m GET -e local_dropped -r 2 [ 1680.630646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.33@tcp 192.168.204.133@tcp [ 1680.653264] LNet: There was an unexpected network error while writing to 192.168.204.133: rc = -22 [ 1680.912806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1680.949394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1680.980585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1681.022672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1681.059628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1681.082388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1681.115648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1681.150626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1681.201691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1681.232597] LNet: 88802:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1681.251543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1681.280838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1681.317510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1681.357836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1681.404085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1681.459648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1682.033567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp -m GET -e local_aborted -r 2 [ 1682.062815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp1 -m GET -e local_aborted -r 2 [ 1682.098533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp -m GET -e local_aborted -r 2 [ 1682.127349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp1 -m GET -e local_aborted -r 2 [ 1682.155914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp -m GET -e local_aborted -r 2 [ 1682.202503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp1 -m GET -e local_aborted -r 2 [ 1682.262751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp -m GET -e local_aborted -r 2 [ 1682.306576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp1 -m GET -e local_aborted -r 2 [ 1682.376598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.33@tcp 192.168.204.133@tcp [ 1682.403819] LNet: There was an unexpected network error while writing to 192.168.204.133: rc = -22 [ 1682.709811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1682.743958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1682.777190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1682.814874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1682.860166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1682.904068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1682.954934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1682.987706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1683.024794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1683.051888] LNet: 88924:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1683.084589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1683.145406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1683.197170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1683.247742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1683.290825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1683.331403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1684.051688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp -m GET -e local_no_route -r 2 [ 1684.100595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp1 -m GET -e local_no_route -r 2 [ 1684.135194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp -m GET -e local_no_route -r 2 [ 1684.171782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp1 -m GET -e local_no_route -r 2 [ 1684.195705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp -m GET -e local_no_route -r 2 [ 1684.235710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp1 -m GET -e local_no_route -r 2 [ 1684.279655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp -m GET -e local_no_route -r 2 [ 1684.325803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp1 -m GET -e local_no_route -r 2 [ 1684.384278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.33@tcp 192.168.204.133@tcp [ 1684.416668] LNet: There was an unexpected network error while writing to 192.168.204.133: rc = -22 [ 1684.701911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1684.744943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1684.801205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1684.839219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1684.881163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1684.920270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1684.971697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1685.027431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1685.079143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1685.095756] LNet: 89046:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1685.115394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1685.178714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1685.216498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1685.252712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1685.282431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1685.307675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1685.822989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp -m GET -e local_timeout -r 2 [ 1685.854847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp1 -m GET -e local_timeout -r 2 [ 1685.879064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp -m GET -e local_timeout -r 2 [ 1685.922819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp1 -m GET -e local_timeout -r 2 [ 1685.996501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp -m GET -e local_timeout -r 2 [ 1686.034516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp1 -m GET -e local_timeout -r 2 [ 1686.072789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp -m GET -e local_timeout -r 2 [ 1686.104650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp1 -m GET -e local_timeout -r 2 [ 1686.141314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.33@tcp 192.168.204.133@tcp [ 1686.435715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1686.477901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1686.526458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1686.558556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1686.589925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1686.622907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1686.672280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1686.699737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1686.732356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1686.761951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1686.789956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1686.816747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1686.847919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1686.883590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1686.919413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1687.454805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp -m GET -e remote_dropped -r 2 [ 1687.482286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp1 -m GET -e remote_dropped -r 2 [ 1687.509306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp -m GET -e remote_dropped -r 2 [ 1687.555500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp1 -m GET -e remote_dropped -r 2 [ 1687.589190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp -m GET -e remote_dropped -r 2 [ 1687.645245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp1 -m GET -e remote_dropped -r 2 [ 1687.707701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp -m GET -e remote_dropped -r 2 [ 1687.752872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp1 -m GET -e remote_dropped -r 2 [ 1687.795427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.33@tcp 192.168.204.133@tcp [ 1687.816118] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.133@tcp [ 1688.071556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1688.116601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1688.147731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1688.176360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1688.205586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1688.235259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1688.275534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1688.318151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1688.370413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1688.420756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1688.462648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1688.499565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1688.529895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1688.560534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1688.592624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1694.015979] LNet: 89365:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1694.024985] LNetError: 89365:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1694.036818] LNet: Removed LNI 192.168.204.33@tcp [ 1694.046246] LNet: Removed LNI 192.168.204.33@tcp1 [ 1694.379193] Key type .llcrypt unregistered [ 1694.386265] Key type ._llcrypt unregistered [ 1712.068976] Key type ._llcrypt registered [ 1712.070735] Key type .llcrypt registered [ 1713.593181] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 13:20:28 (1788542428) [ 1713.972658] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1713.987598] alg: No test for adler32 (adler32-zlib) [ 1714.888329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1714.962962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1714.999187] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1715.923980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1715.954646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.33@tcp [ 1715.973518] LNetError: 90341:0:(api-ni.c:4483:lnet_ni_set_healthv()) manually adding local NI 192.168.204.33@tcp to recovery [ 1716.054510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.204.33@tcp [ 1716.161287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1722.602943] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 13:20:38 (1788542438) [ 1722.822492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1722.876761] LNet: 90930:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1722.893837] LNetError: 90930:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1722.928109] LNet: Removed LNI 192.168.204.33@tcp [ 1722.950428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1722.995894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1723.025700] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1724.007351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1724.045984] LNet: Added LNI 192.168.204.33@tcp1 [8/256/0/180] [ 1725.168906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.204.33@tcp [ 1725.192670] LNetError: 91196:0:(api-ni.c:4483:lnet_ni_set_healthv()) manually adding local NI 192.168.204.33@tcp to recovery [ 1726.049510] LNet: 1 local NIs in recovery (showing 1): 192.168.204.33@tcp [ 1726.050803] LNet: There was an unexpected network error while writing to 192.168.204.33: rc = -22 [ 1726.466708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.204.33@tcp [ 1735.377704] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 13:20:50 (1788542450) [ 1737.573362] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg433-server [ 1739.500103] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 13:20:55 (1788542455) [ 1741.595443] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg433-server [ 1743.155087] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 13:20:58 (1788542458) [ 1744.940645] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg433-server [ 1746.294067] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 13:21:02 (1788542462) [ 1746.684518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1746.709274] LNet: 92826:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1746.718919] LNetError: 92826:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1746.736353] LNet: Removed LNI 192.168.204.33@tcp [ 1746.756147] LNet: Removed LNI 192.168.204.33@tcp1 [ 1746.796503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1746.859413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1746.896792] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1747.544838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1747.613801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1747.667843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1747.768603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1747.825471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1747.937913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1754.605714] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 13:21:09 (1788542469) [ 1754.724214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1754.784715] LNet: 93551:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1754.801210] LNetError: 93551:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1754.808333] LNetError: 93551:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1754.832395] LNet: Removed LNI 192.168.204.33@tcp [ 1754.861888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1754.957936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1754.973378] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1763.450673] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 13:21:19 (1788542479) [ 1763.561080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1763.594237] LNet: 94257:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1763.600640] LNetError: 94257:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1763.627127] LNet: Removed LNI 192.168.204.33@tcp [ 1763.660283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1763.696729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1763.714249] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1770.424419] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 13:21:26 (1788542486) [ 1772.478314] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1774.546266] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 13:21:29 (1788542489) [ 1775.372457] LNet: 95296:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1775.388896] LNetError: 95296:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1775.409464] LNet: Removed LNI 192.168.204.33@tcp [ 1776.114139] Key type .llcrypt unregistered [ 1776.118197] Key type ._llcrypt unregistered [ 1783.028917] Key type ._llcrypt registered [ 1783.035486] Key type .llcrypt registered [ 1783.452762] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1783.478369] alg: No test for adler32 (adler32-zlib) [ 1784.549842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1784.615492] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1792.807295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.133@tcp [ 1799.259331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.204.133@tcp [ 1799.886989] LNet: 95711:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1799.900054] LNetError: 95711:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1799.939784] LNet: Removed LNI 192.168.204.33@tcp [ 1800.462387] Key type .llcrypt unregistered [ 1800.464257] Key type ._llcrypt unregistered [ 1812.072738] Key type ._llcrypt registered [ 1812.076651] Key type .llcrypt registered [ 1813.310682] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1815.266917] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1817.084953] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 13:22:12 (1788542532) [ 1818.814882] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1820.697364] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 13:22:16 (1788542536) [ 1822.574414] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg433-server [ 1824.313656] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 13:22:20 (1788542540) [ 1826.283108] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg433-server [ 1827.815318] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 13:22:23 (1788542543) [ 1829.426650] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1829.460717] alg: No test for adler32 (adler32-zlib) [ 1830.613657] Lustre: Lustre: Build Version: 2.17.58_2_g4093f3c [ 1830.992782] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1832.679179] Key type lgssc registered [ 1834.068031] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1855.502168] Key type lgssc unregistered [ 1855.793758] LNet: 98882:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1855.799536] LNetError: 98882:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1855.828714] LNet: Removed LNI 192.168.204.33@tcp [ 1856.522478] Key type .llcrypt unregistered [ 1856.524327] Key type ._llcrypt unregistered [ 1873.182719] Key type ._llcrypt registered [ 1873.184590] Key type .llcrypt registered [ 1874.620990] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 13:23:10 (1788542590) [ 1875.113433] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1875.142516] alg: No test for adler32 (adler32-zlib) [ 1876.138836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1876.254923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1876.321043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1876.413116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1876.500277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1883.972181] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 13:23:19 (1788542599) [ 1884.110039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1884.143536] LNet: 100288:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1884.181847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1884.240089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1884.277584] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1885.239322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1885.286712] LNet: 100439:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1885.298811] LNetError: 100439:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1885.325207] LNet: Removed LNI 192.168.204.33@tcp [ 1885.349453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1885.426778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1885.453702] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1885.978632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1885.998706] LNetError: 100467:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1886.014735] LNet: Removed LNI 192.168.204.33@tcp [ 1886.032527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1886.093707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1886.136573] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1892.941099] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 13:23:28 (1788542608) [ 1893.083982] LNet: 101057:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1893.092111] LNet: 101057:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1893.100819] LNetError: 101057:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1893.125814] LNet: Removed LNI 192.168.204.33@tcp [ 1893.857844] Key type .llcrypt unregistered [ 1893.859446] Key type ._llcrypt unregistered [ 1899.627460] Key type ._llcrypt registered [ 1899.629413] Key type .llcrypt registered [ 1899.847087] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1899.887716] alg: No test for adler32 (adler32-zlib) [ 1900.893324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1900.998369] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1901.001048] LNet: *** cfs_fail_loc=e003, val=100*** [ 1901.830881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 1901.859650] LNet: *** cfs_fail_loc=e003, val=100*** [ 1901.862951] LNet: Skipped 2 previous similar messages [ 1901.878420] Ping reply received with mapping: [ 1901.880087] found metadata reply NI[192.168.204.33@tcp] [ 1901.884014] 21 cc a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1901.888097] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1901.893680] 5a 5a 5a 5a 5a 5a 5a 5a [ 1902.153808] LNet: 101452:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1902.171746] LNetError: 101452:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1902.199902] LNet: Removed LNI 192.168.204.33@tcp [ 1902.789205] Key type .llcrypt unregistered [ 1902.791807] Key type ._llcrypt unregistered [ 1917.172286] Key type ._llcrypt registered [ 1917.180158] Key type .llcrypt registered [ 1918.959221] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 13:23:54 (1788542634) [ 1919.665431] Key type .llcrypt unregistered [ 1919.668342] Key type ._llcrypt unregistered [ 1926.862558] Key type ._llcrypt registered [ 1926.866033] Key type .llcrypt registered [ 1927.112519] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1927.126729] alg: No test for adler32 (adler32-zlib) [ 1928.250187] LNetError: 102489:0:(api-ni.c:2855:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1928.673541] Key type .llcrypt unregistered [ 1928.675845] Key type ._llcrypt unregistered [ 1934.486708] Key type ._llcrypt registered [ 1934.488266] Key type .llcrypt registered [ 1935.836503] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 13:24:11 (1788542651) [ 1936.499526] Key type .llcrypt unregistered [ 1936.501435] Key type ._llcrypt unregistered [ 1943.066738] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1943.071744] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1945.692466] Key type ._llcrypt registered [ 1945.694487] Key type .llcrypt registered [ 1945.829302] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1945.842192] alg: No test for adler32 (adler32-zlib) [ 1946.798572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1946.867587] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1957.296846] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 13:24:32 (1788542672) [ 1957.407364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1957.456475] LNet: 104254:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1957.460927] LNetError: 104254:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1957.491104] LNet: Removed LNI 10.1.2.3@tcp [ 1957.522182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1958.514451] Key type .llcrypt unregistered [ 1958.524556] Key type ._llcrypt unregistered [ 1965.729664] Key type ._llcrypt registered [ 1965.733673] Key type .llcrypt registered [ 1966.021813] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1966.054312] alg: No test for adler32 (adler32-zlib) [ 1967.136119] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1967.262803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 1967.423209] LNet: 104548:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1967.430990] LNetError: 104548:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1967.449189] LNet: Removed LNI 192.168.204.33@tcp [ 1968.279500] Key type .llcrypt unregistered [ 1968.281184] Key type ._llcrypt unregistered [ 1978.339293] Key type ._llcrypt registered [ 1978.341560] Key type .llcrypt registered [ 1979.694941] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 13:24:55 (1788542695) [ 1980.446323] Key type .llcrypt unregistered [ 1980.448242] Key type ._llcrypt unregistered [ 1986.574754] Key type ._llcrypt registered [ 1986.577793] Key type .llcrypt registered [ 1986.999947] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1987.041336] alg: No test for adler32 (adler32-zlib) [ 1996.174714] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 13:25:11 (1788542711) [ 1996.455924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1996.513273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1996.615215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1996.639992] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1997.799622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1997.918322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1997.987178] LNet: Removed LNI 192.168.204.33@tcp [ 1998.479612] LNet: 106286:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1999.039254] Key type .llcrypt unregistered [ 1999.040694] Key type ._llcrypt unregistered [ 2011.786551] Key type ._llcrypt registered [ 2011.788208] Key type .llcrypt registered [ 2013.324233] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 13:25:28 (1788542728) [ 2013.762723] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2013.773810] alg: No test for adler32 (adler32-zlib) [ 2014.638937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2014.735239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2014.768933] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2015.707343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 2015.790601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.204.33@tcp [ 2015.834104] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2015.840799] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2015.856735] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2015, 0, 0 [ 2015.865826] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2015.873711] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2015.882041] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2015.888843] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2015, 0, 0 [ 2015.894908] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2015.898821] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2015.907307] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2015.909896] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2015, 0, 2065 [ 2015.915895] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2015.918934] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2015.928923] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2015.934819] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2015, 0, 2065 [ 2015.944898] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2022.445129] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 13:25:38 (1788542738) [ 2022.937943] LNet: 107823:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2022.947318] LNetError: 107823:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2022.958467] LNet: Removed LNI 192.168.204.33@tcp [ 2023.765383] Key type .llcrypt unregistered [ 2023.767266] Key type ._llcrypt unregistered [ 2029.843853] Key type ._llcrypt registered [ 2029.851113] Key type .llcrypt registered [ 2030.106809] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2030.121854] alg: No test for adler32 (adler32-zlib) [ 2031.140815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2031.172961] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2039.115300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.133@tcp [ 2047.170882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2047.213603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2047.240555] LNet: Added LNI 192.168.204.33@tcp1 [8/256/0/180] [ 2047.915270] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2048.003629] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2048.012273] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2048.019198] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2050.428399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2050.474625] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2062.942191] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 13:26:17 (1788542777) [ 2063.074685] LNet: 109321:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2063.090843] LNetError: 109321:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2063.109511] LNet: Removed LNI 192.168.204.33@tcp [ 2064.159404] LNet: Removed LNI 192.168.204.33@tcp1 [ 2064.168204] LNetError: 109321:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2064.185536] LNetError: 109321:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2064.210489] LNet: Removed LNI 10.1.2.3@tcp99 [ 2065.188421] Key type .llcrypt unregistered [ 2065.194323] Key type ._llcrypt unregistered [ 2072.869944] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2072.877287] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2075.488147] Key type ._llcrypt registered [ 2075.493933] Key type .llcrypt registered [ 2075.786607] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2075.825437] alg: No test for adler32 (adler32-zlib) [ 2076.733195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2076.807191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2076.859506] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2077.914944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2077.958961] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2081.340974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2081.395535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2081.504931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2081.569962] LNetError: 110070:0:(peer.c:2366:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2081.646517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2081.723394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.33@tcp [ 2082.359146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2082.445611] LNet: 110091:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2082.453851] LNetError: 110091:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2082.468788] LNet: Removed LNI 10.1.2.3@tcp [ 2083.497456] LNet: Removed LNI 192.168.204.33@tcp [ 2084.336418] Key type .llcrypt unregistered [ 2084.337888] Key type ._llcrypt unregistered [ 2095.089107] Key type ._llcrypt registered [ 2095.090545] Key type .llcrypt registered [ 2096.369717] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 13:26:52 (1788542812) [ 2096.866960] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2096.902967] alg: No test for adler32 (adler32-zlib) [ 2097.856451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2097.953744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2097.980645] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2098.900420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 2099.033759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg433-client@tcp [ 2106.186232] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 13:27:01 (1788542821) [ 2106.307296] LNet: 111615:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2106.324991] LNetError: 111615:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2106.343457] LNet: Removed LNI 192.168.204.33@tcp [ 2107.088745] Key type .llcrypt unregistered [ 2107.095420] Key type ._llcrypt unregistered [ 2114.227699] Key type ._llcrypt registered [ 2114.229846] Key type .llcrypt registered [ 2114.455208] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2114.468700] alg: No test for adler32 (adler32-zlib) [ 2115.444874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2115.486951] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2115.528625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2122.162288] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 13:27:17 (1788542837) [ 2123.504394] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2125.089448] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 13:27:20 (1788542840) [ 2126.594307] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2128.550057] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 13:27:24 (1788542844) [ 2128.791052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2128.833152] LNet: 113042:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2128.845810] LNetError: 113042:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2128.865087] LNet: Removed LNI 192.168.204.33@tcp [ 2128.888938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2129.012282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2129.049776] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2129.910306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2129.976297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2137.858489] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 13:27:33 (1788542853) [ 2138.256643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2138.286655] LNet: 113751:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2138.306233] LNetError: 113751:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2138.326799] LNet: Removed LNI 192.168.204.33@tcp [ 2138.350550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2138.423759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2138.475731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2139.253215] Key type .llcrypt unregistered [ 2139.257431] Key type ._llcrypt unregistered [ 2152.698300] Key type ._llcrypt registered [ 2152.704158] Key type .llcrypt registered [ 2154.213466] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 13:27:49 (1788542869) [ 2154.772790] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2154.805380] alg: No test for adler32 (adler32-zlib) [ 2155.837489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2155.893350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2155.936238] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2156.759909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2156.816903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.33@tcp [ 2164.334846] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 13:27:59 (1788542879) [ 2164.429517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2164.468157] LNet: 115296:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2164.484184] LNetError: 115296:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2164.517596] LNet: Removed LNI 192.168.204.33@tcp [ 2164.562000] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2164.665123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2164.693260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2171.285679] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 13:28:07 (1788542887) [ 2171.584677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2171.623600] LNet: 115884:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2171.642811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2171.758915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2172.133374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2172.178395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2172.275926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2172.702383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2172.750666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2172.867533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2172.878699] LNet: 115917:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2172.910747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2172.968026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2173.604933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2173.630750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2173.680835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2174.088039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2174.122214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2180.329916] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 13:28:15 (1788542895) [ 2180.496088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2180.556862] LNet: 116512:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2180.617125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2180.736636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2180.803281] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2181.899812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2181.943490] LNet: Added LNI 192.168.204.33@tcp2 [8/256/0/180] [ 2183.499665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2183.679723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.33@tcp [ 2184.124470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2184.168658] LNetError: 116798:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2184.189737] LNet: Removed LNI 192.168.204.33@tcp [ 2184.217394] LNet: Removed LNI 192.168.204.33@tcp2 [ 2184.305981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2184.442960] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2184.463799] LNet: Added LNI 192.168.204.33@tcp2 [8/256/0/180] [ 2184.480976] LNet: 116804:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2185.591483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.33@tcp [ 2186.062805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2186.168631] LNet: 116841:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2186.174346] LNet: 116841:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2186.186848] LNetError: 116841:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2186.202925] LNetError: 116841:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2186.216725] LNet: Removed LNI 192.168.204.33@tcp [ 2186.243791] LNet: Removed LNI 192.168.204.33@tcp2 [ 2186.272302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2186.389101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2186.439386] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2187.421579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2187.436259] LNet: Added LNI 192.168.204.33@tcp2 [8/256/0/180] [ 2188.564940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.33@tcp [ 2188.616963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2188.683586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.33@tcp [ 2188.993986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2189.068096] LNetError: 117126:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2189.074677] LNetError: 117126:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2189.086909] LNet: Removed LNI 192.168.204.33@tcp [ 2190.120800] LNet: Removed LNI 192.168.204.33@tcp2 [ 2190.272379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2190.429911] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2190.441085] LNet: Added LNI 192.168.204.33@tcp2 [8/256/0/180] [ 2190.456511] LNet: 117132:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2192.005988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.33@tcp [ 2192.377211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2192.437244] LNet: Removed LNI 192.168.204.33@tcp [ 2193.517266] LNet: Removed LNI 192.168.204.33@tcp2 [ 2193.553966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2193.615901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2193.656280] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2194.701615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2194.732096] LNet: Added LNI 192.168.204.33@tcp2 [8/256/0/180] [ 2196.040566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2196.143704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.33@tcp [ 2196.510204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2196.559489] LNet: 117449:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2196.573520] LNet: 117449:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2196.586184] LNetError: 117449:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2196.599289] LNetError: 117449:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2196.610200] LNet: Removed LNI 192.168.204.33@tcp [ 2196.626854] LNet: Removed LNI 192.168.204.33@tcp2 [ 2196.702663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2196.831995] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2196.846794] LNet: Added LNI 192.168.204.33@tcp2 [8/256/0/180] [ 2196.852635] LNet: 117455:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2197.537303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.33@tcp [ 2197.912705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2199.010386] LNet: Removed LNI 192.168.204.33@tcp [ 2199.031074] LNet: Removed LNI 192.168.204.33@tcp2 [ 2199.122978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2199.245916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2199.272856] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2200.329965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2200.367410] LNet: Added LNI 192.168.204.33@tcp2 [8/256/0/180] [ 2201.171497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.33@tcp [ 2201.215682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2201.336145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.33@tcp [ 2201.652521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2202.719179] LNet: Removed LNI 192.168.204.33@tcp [ 2203.749753] LNet: Removed LNI 192.168.204.33@tcp2 [ 2203.817314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2203.945572] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2204.004541] LNet: Added LNI 192.168.204.33@tcp2 [8/256/0/180] [ 2204.015434] LNet: 117784:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2204.530659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.33@tcp [ 2204.682654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2204.709986] LNetError: 117821:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2204.714489] LNetError: 117821:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2204.719991] LNet: Removed LNI 192.168.204.33@tcp [ 2205.737481] LNet: Removed LNI 192.168.204.33@tcp2 [ 2205.788666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2205.913526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2205.975045] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2206.579470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2206.592476] LNet: Added LNI 192.168.204.33@tcp2 [8/256/0/180] [ 2207.346167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2207.391808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2207.456776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.33@tcp [ 2207.761487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2208.799496] LNet: Removed LNI 192.168.204.33@tcp [ 2208.828409] LNet: Removed LNI 192.168.204.33@tcp2 [ 2208.887757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2208.942961] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2208.950193] LNet: Added LNI 192.168.204.33@tcp2 [8/256/0/180] [ 2208.956194] LNet: 118125:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2209.282573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.33@tcp [ 2209.508737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2210.527269] LNet: Removed LNI 192.168.204.33@tcp [ 2210.537831] LNet: Removed LNI 192.168.204.33@tcp2 [ 2210.581099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2210.659438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2210.697219] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2211.171354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2211.182227] LNet: Added LNI 192.168.204.33@tcp2 [8/256/0/180] [ 2211.778691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.33@tcp [ 2211.817463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2211.857451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2211.914208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.33@tcp [ 2212.161266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2213.215151] LNet: Removed LNI 192.168.204.33@tcp [ 2214.248175] LNet: Removed LNI 192.168.204.33@tcp2 [ 2214.305075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2214.393062] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2214.400178] LNet: Added LNI 192.168.204.33@tcp2 [8/256/0/180] [ 2214.733409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.33@tcp [ 2222.403675] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2224.312935] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 13:28:59 (1788542939) [ 2224.688608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2224.726730] LNet: 119244:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2224.738956] LNet: 119244:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 6 previous similar messages [ 2224.750447] LNetError: 119244:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2224.757356] LNetError: 119244:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2224.768271] LNet: Removed LNI 192.168.204.33@tcp [ 2225.841858] LNet: Removed LNI 192.168.204.33@tcp2 [ 2225.873749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2226.132582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2226.152100] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2226.827792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2226.853822] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2227.865837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2227.891545] LNet: Removed LNI 10.1.2.3@tcp [ 2227.901705] LNet: Removed LNI 192.168.204.33@tcp [ 2227.938885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2228.020061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2228.055911] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2228.065049] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2228.535326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2228.569838] LNet: Removed LNI 192.168.204.33@tcp [ 2228.585891] LNet: Removed LNI 10.1.2.3@tcp [ 2235.096662] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 13:29:10 (1788542950) [ 2235.416605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2235.451337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2235.500952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2235.541473] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2236.478693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2236.528252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2236.577734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 2236.637593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 2236.711650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 2236.756802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 2236.816530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 2236.851767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 2236.905556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 2236.951914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 2237.009326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 2237.062903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 2237.308476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2243.323461] Lustre: DEBUG MARKER: == sanity-lnet test 476: lnetctl set latency_stats toggles capture ========================================================== 13:29:19 (1788542959) [ 2243.592611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2244.710473] LNet: Removed LNI 192.168.204.33@tcp [ 2244.745480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2244.816947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2244.847553] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2245.531496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 0 [ 2245.566894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2245.607920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 2245.645385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 2245.672620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 2245.723316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 2245.745699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 2245.771873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 2245.797794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 2245.830269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 2245.860772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 2245.893190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 2245.962613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2245.990852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 2246.018358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 2246.042825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 2246.081659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 2246.136521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 2246.197509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 2246.253755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 2246.312588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 2246.371585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 2246.441154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 2252.516434] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 13:29:28 (1788542968) [ 2252.635435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2253.734739] LNet: Removed LNI 192.168.204.33@tcp [ 2253.776605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2253.929940] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2254.143639] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2254.150705] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2254.157537] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2256.561165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2256.589349] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2257.474575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2257.501528] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2260.238438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.33@tcp [ 2260.282507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2260.404298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.204.33@tcp -m PUT -e local_timeout -r 1 [ 2260.544353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2260.634542] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2260.639920] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2261.063333] LNetError: 121679:0:(lib-move.c:4794:lnet_parse()) 192.168.204.33@tcp, src 192.168.204.33@tcp: Dropping PUT (error -108 looking up sender) [ 2262.111757] LNet: 122139:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2262.125268] LNet: 122139:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2262.134610] LNetError: 122139:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2262.155889] LNetError: 122139:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2262.164105] LNet: Removed LNI 10.1.2.3@tcp [ 2263.216558] LNet: Removed LNI 192.168.204.33@tcp [ 2263.647485] Key type .llcrypt unregistered [ 2263.649356] Key type ._llcrypt unregistered [ 2276.798926] Key type ._llcrypt registered [ 2276.803940] Key type .llcrypt registered [ 2278.245809] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 13:29:53 (1788542993) [ 2278.618765] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2278.636974] alg: No test for adler32 (adler32-zlib) [ 2279.643235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2280.206773] LNet: 123001:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2280.907466] Key type .llcrypt unregistered [ 2280.910591] Key type ._llcrypt unregistered [ 2287.586403] Key type ._llcrypt registered [ 2287.591462] Key type .llcrypt registered [ 2287.809267] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2287.825298] alg: No test for adler32 (adler32-zlib) [ 2288.832528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2288.880976] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2297.374793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.133@tcp [ 2297.586217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2298.252770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.133@tcp [ 2298.572240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2298.605341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2298.636630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2298.665484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2298.691573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2298.725356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2298.762321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2298.804661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2298.849504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2298.869358] LNet: 123451:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2298.899705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2298.931363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2298.968721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2299.014746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2299.067827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2299.094620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2299.906638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2300.146561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2300.189265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2300.203244] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.133@tcp [ 2300.239674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2300.285385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2300.322330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2300.358174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2300.398669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2300.443550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2300.502744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2300.534079] LNet: 123570:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2300.563650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2300.621264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2300.681541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2300.741337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2300.798744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2300.863706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2301.715817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.133@tcp [ 2302.025306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2302.056445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2302.092598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2302.128640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2302.168769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2302.211742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2302.261779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2302.318785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2302.399878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2302.434272] LNet: 123690:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2302.469308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2302.521739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2302.568504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2302.634654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2302.687959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2302.723148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2303.484213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2303.821994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2303.852258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2303.888410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2303.943943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2303.980300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2304.004254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2304.022188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2304.041472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2304.080685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2304.109190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2304.135384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2304.163149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2304.208140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2304.254355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2304.305619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2304.685950] LNet: 123855:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2304.693329] LNetError: 123855:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2304.711325] LNet: Removed LNI 192.168.204.33@tcp [ 2305.153833] Key type .llcrypt unregistered [ 2305.155062] Key type ._llcrypt unregistered [ 2320.871294] Key type ._llcrypt registered [ 2320.873242] Key type .llcrypt registered [ 2322.036540] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 13:30:37 (1788543037) [ 2322.237265] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2322.248593] alg: No test for adler32 (adler32-zlib) [ 2323.204681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2323.670749] LNet: 124740:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2324.520429] Key type .llcrypt unregistered [ 2324.523538] Key type ._llcrypt unregistered [ 2330.318154] Key type ._llcrypt registered [ 2330.324435] Key type .llcrypt registered [ 2330.587187] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2330.617426] alg: No test for adler32 (adler32-zlib) [ 2331.577979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2331.614743] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2340.305804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.133@tcp [ 2350.941954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2351.032808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2351.069155] LNet: Added LNI 192.168.204.33@tcp1 [8/256/0/180] [ 2351.961783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2352.650683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.133@tcp [ 2352.940125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2352.989847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2353.067615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2353.132884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2353.174318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2353.194781] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.133@tcp [ 2353.208685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2353.238653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2353.268659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2353.298085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2353.317791] LNet: 125393:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2353.344290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2353.386252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2353.418704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2353.476727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2353.536629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2353.574615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2354.352470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2354.709970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2354.749321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2354.781434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2354.812391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2354.845260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2354.874224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2354.906635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2354.938094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2354.974268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2354.990430] LNet: 125512:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2355.004703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2355.047435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2355.094743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2355.140199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2355.180107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2355.215279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2356.006514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.133@tcp [ 2356.266392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2356.301764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2356.334845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2356.373398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2356.410880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2356.458792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2356.516438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2356.578158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2356.630442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2356.660979] LNet: 125631:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2356.699542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2356.754936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2356.797684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2356.846973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2356.876355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2356.905884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2357.756477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2358.182701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2358.257436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2358.300353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2358.358762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2358.398599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2358.432309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2358.483249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2358.526910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2358.572497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2358.616394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2358.664464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2358.732625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2358.817601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2358.864794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2358.927095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2364.626911] LNet: 125832:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2364.634469] LNetError: 125832:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2364.641353] LNet: Removed LNI 192.168.204.33@tcp [ 2364.651689] LNet: Removed LNI 192.168.204.33@tcp1 [ 2365.176791] Key type .llcrypt unregistered [ 2365.178510] Key type ._llcrypt unregistered [ 2384.764855] Key type ._llcrypt registered [ 2384.767530] Key type .llcrypt registered [ 2386.306329] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 13:31:41 (1788543101) [ 2387.001656] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2387.018154] alg: No test for adler32 (adler32-zlib) [ 2388.104460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2388.175290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2388.208556] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2389.259647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2389.289957] LNet: Added LNI 192.168.204.33@tcp2 [8/256/0/180] [ 2390.476687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.33@tcp [ 2390.575889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2391.647210] LNet: Removed LNI 192.168.204.33@tcp [ 2398.826392] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 13:31:54 (1788543114) [ 2400.548788] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2402.761287] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 13:31:58 (1788543118) [ 2411.397271] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 13:32:06 (1788543126) [ 2419.791739] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 13:32:15 (1788543135) [ 2428.057628] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 13:32:23 (1788543143) [ 2436.031975] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 13:32:31 (1788543151) [ 2445.335732] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 13:32:40 (1788543160) [ 2453.708874] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 13:32:49 (1788543169) [ 2462.672055] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 13:32:58 (1788543178) [ 2471.899758] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 13:33:07 (1788543187) [ 2479.954437] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 13:33:15 (1788543195) [ 2489.339029] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 13:33:24 (1788543204) [ 2498.136767] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 13:33:33 (1788543213) [ 2507.142612] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 13:33:42 (1788543222) [ 2515.152485] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 13:33:50 (1788543230) [ 2523.809383] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 13:33:59 (1788543239) [ 2532.811485] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 13:34:08 (1788543248) [ 2542.567962] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 13:34:17 (1788543257) [ 2551.874680] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 13:34:27 (1788543267) [ 2552.004887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2552.051754] LNet: 137524:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2552.065745] LNetError: 137524:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2552.125628] LNet: Removed LNI 192.168.204.33@tcp2 [ 2552.161901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2552.370423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2552.394706] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2553.789777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2553.823080] LNet: 137674:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2553.836375] LNetError: 137674:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2553.857579] LNet: Removed LNI 192.168.204.33@tcp [ 2553.886971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2553.962215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2553.993724] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2561.050497] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 13:34:36 (1788543276) [ 2569.559282] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 13:34:45 (1788543285) [ 2578.116318] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 13:34:53 (1788543293) [ 2586.878974] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 13:35:02 (1788543302) [ 2594.582772] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 13:35:09 (1788543309) [ 2602.383863] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 13:35:17 (1788543317) [ 2611.679677] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 13:35:27 (1788543327) [ 2620.292183] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 13:35:35 (1788543335) [ 2629.616179] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 13:35:44 (1788543344) [ 2637.501545] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 13:35:53 (1788543353) [ 2645.204843] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 13:36:00 (1788543360) [ 2652.310737] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 13:36:07 (1788543367) [ 2659.250921] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2419 sec ========== 13:36:14 (1788543374) [ 2659.707319] LNet: 145140:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2659.714734] LNetError: 145140:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2659.727170] LNet: Removed LNI 192.168.204.33@tcp [ 2660.474706] Key type .llcrypt unregistered [ 2660.479865] Key type ._llcrypt unregistered [ 2674.755243] Key type ._llcrypt registered [ 2674.765131] Key type .llcrypt registered [ 2675.475239] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2675.491788] alg: No test for adler32 (adler32-zlib) [ 2676.622380] Lustre: Lustre: Build Version: 2.17.58_2_g4093f3c [ 2677.015972] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2678.783530] Key type lgssc registered [ 2680.445144] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2754.558360] Lustre: Mounted lustre-client - version 2.17.58_2_g4093f3c [ 2760.655484] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2770.615980] Lustre: Unmounted lustre-client [ 2792.265436] Key type lgssc unregistered [ 2792.646927] LNet: 148090:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2792.664049] LNetError: 148090:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2792.685972] LNet: Removed LNI 192.168.204.33@tcp [ 2793.579239] Key type .llcrypt unregistered [ 2793.581817] Key type ._llcrypt unregistered