[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 456305157 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.002340] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004013] kvm-guest: setup PV IPIs [ 0.007666] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008035] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.010124] LSM: Security Framework initializing [ 0.011058] Yama: becoming mindful. [ 0.012031] SELinux: Initializing. [ 0.013057] *** VALIDATE selinux *** [ 0.021600] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026350] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027150] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028095] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029120] *** VALIDATE tmpfs *** [ 0.031331] *** VALIDATE proc *** [ 0.032249] *** VALIDATE cgroup *** [ 0.033009] *** VALIDATE cgroup2 *** [ 0.035005] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036162] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038030] Spectre V2 : User space: Vulnerable [ 0.039007] Speculative Store Bypass: Vulnerable [ 0.042500] debug: unmapping init [mem 0xffffffffa0059000-0xffffffffa0060fff] [ 0.044772] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045520] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046019] ... version: 2 [ 0.047013] ... bit width: 48 [ 0.048009] ... generic registers: 4 [ 0.048890] ... value mask: 0000ffffffffffff [ 0.049010] ... max period: 00007fffffffffff [ 0.050010] ... fixed-purpose events: 3 [ 0.050888] ... event mask: 000000070000000f [ 0.051234] rcu: Hierarchical SRCU implementation. [ 0.053370] smp: Bringing up secondary CPUs ... [ 0.054550] x86: Booting SMP configuration: [ 0.055023] .... node #0, CPUs: #1 #2 #3 [ 0.058072] smp: Brought up 1 node, 4 CPUs [ 0.060014] smpboot: Max logical packages: 1 [ 0.061011] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.146099] node 0 deferred pages initialised in 82ms [ 0.148231] devtmpfs: initialized [ 0.149216] x86/mm: Memory block size: 128MB [ 0.151719] gcov: version magic: 0x41383552 [ 0.153218] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.154102] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.155291] pinctrl core: initialized pinctrl subsystem [ 0.156177] [ 0.156695] ************************************************************* [ 0.157017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.158016] ** ** [ 0.159011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.160013] ** ** [ 0.161014] ** This means that this kernel is built to expose internal ** [ 0.162013] ** IOMMU data structures, which may compromise security on ** [ 0.163010] ** your system. ** [ 0.164014] ** ** [ 0.165011] ** If you see this message and you are not debugging the ** [ 0.166011] ** kernel, report this immediately to your vendor! ** [ 0.167017] ** ** [ 0.168013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.169011] ************************************************************* [ 0.170679] NET: Registered protocol family 16 [ 0.171425] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.172053] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.173075] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.174417] cpuidle: using governor menu [ 0.177356] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.178423] PCI: Using configuration type 1 for base access [ 0.179135] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.186051] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.188032] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.192033] cryptd: max_cpu_qlen set to 1000 [ 0.193224] ACPI: Added _OSI(Module Device) [ 0.194011] ACPI: Added _OSI(Processor Device) [ 0.195016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.196012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.200014] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.204162] ACPI: Interpreter enabled [ 0.205054] ACPI: PM: (supports S0 S3 S4 S5) [ 0.206012] ACPI: Using IOAPIC for interrupt routing [ 0.207109] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.208375] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.217187] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.218039] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.219019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.220080] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.222318] acpiphp: Slot [2] registered [ 0.223118] acpiphp: Slot [5] registered [ 0.224092] acpiphp: Slot [6] registered [ 0.225088] acpiphp: Slot [3] registered [ 0.226074] acpiphp: Slot [4] registered [ 0.227077] acpiphp: Slot [7] registered [ 0.228096] acpiphp: Slot [8] registered [ 0.229110] acpiphp: Slot [9] registered [ 0.230096] acpiphp: Slot [10] registered [ 0.231110] acpiphp: Slot [11] registered [ 0.232088] acpiphp: Slot [12] registered [ 0.233105] acpiphp: Slot [13] registered [ 0.234099] acpiphp: Slot [14] registered [ 0.235160] acpiphp: Slot [15] registered [ 0.236119] acpiphp: Slot [16] registered [ 0.237092] acpiphp: Slot [17] registered [ 0.238094] acpiphp: Slot [18] registered [ 0.239093] acpiphp: Slot [19] registered [ 0.240100] acpiphp: Slot [20] registered [ 0.241090] acpiphp: Slot [21] registered [ 0.242114] acpiphp: Slot [22] registered [ 0.243138] acpiphp: Slot [23] registered [ 0.244116] acpiphp: Slot [24] registered [ 0.245098] acpiphp: Slot [25] registered [ 0.246109] acpiphp: Slot [26] registered [ 0.247110] acpiphp: Slot [27] registered [ 0.248140] acpiphp: Slot [28] registered [ 0.250117] acpiphp: Slot [29] registered [ 0.251109] acpiphp: Slot [30] registered [ 0.253121] acpiphp: Slot [31] registered [ 0.255103] PCI host bridge to bus 0000:00 [ 0.256017] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.259027] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.261027] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.264024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.267025] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.270034] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.272247] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.275074] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.278442] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.286016] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.290152] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.294022] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.296018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.299018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.302067] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.306135] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.309053] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.312860] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.317013] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.327778] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.332859] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.339110] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.357020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.373016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.419020] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.438668] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.445014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.451018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.484013] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.499000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.501258] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.503249] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.504188] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.506063] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.510027] iommu: Default domain type: Passthrough [ 0.512413] SCSI subsystem initialized [ 0.514142] ACPI: bus type USB registered [ 0.515120] usbcore: registered new interface driver usbfs [ 0.517072] usbcore: registered new interface driver hub [ 0.518064] usbcore: registered new device driver usb [ 0.520219] pps_core: LinuxPPS API ver. 1 registered [ 0.522010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.524060] PTP clock support registered [ 0.527041] EDAC MC: Ver: 3.0.0 [ 0.529122] PCI: Using ACPI for IRQ routing [ 0.530783] NetLabel: Initializing [ 0.532012] NetLabel: domain hash size = 128 [ 0.533011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.535082] NetLabel: unlabeled traffic allowed by default [ 0.537279] vgaarb: loaded [ 0.539026] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.540010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.546206] clocksource: Switched to clocksource kvm-clock [ 0.658139] VFS: Disk quotas dquot_6.6.0 [ 0.659361] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.661089] *** VALIDATE ramfs *** [ 0.661850] *** VALIDATE hugetlbfs *** [ 0.662897] pnp: PnP ACPI init [ 0.664816] pnp: PnP ACPI: found 6 devices [ 0.680062] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.683503] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.685830] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.688124] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.690435] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.692674] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.695620] NET: Registered protocol family 2 [ 0.698410] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.702169] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.704428] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.708263] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.710886] TCP: Hash tables configured (established 65536 bind 65536) [ 0.713248] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.715326] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.717224] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.718960] NET: Registered protocol family 1 [ 0.721416] RPC: Registered named UNIX socket transport module. [ 0.723348] RPC: Registered udp transport module. [ 0.724317] RPC: Registered tcp transport module. [ 0.725296] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.726588] NET: Registered protocol family 44 [ 0.727623] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.728917] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.730176] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.731343] PCI: CLS 0 bytes, default 64 [ 0.732270] Unpacking initramfs... [ 2.102799] debug: unmapping init [mem 0xffff8a263cc64000-0xffff8a263ffcffff] [ 2.106609] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.108521] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.111177] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.585829] Initialise system trusted keyrings [ 2.587540] Key type blacklist registered [ 2.589264] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.597899] zbud: loaded [ 2.602652] *** VALIDATE nfs *** [ 2.603612] *** VALIDATE nfs4 *** [ 2.604563] pstore: using deflate compression [ 2.606941] Platform Keyring initialized [ 2.682072] NET: Registered protocol family 38 [ 2.683372] Key type asymmetric registered [ 2.684467] Asymmetric key parser 'x509' registered [ 2.685680] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.687586] io scheduler mq-deadline registered [ 2.688576] io scheduler kyber registered [ 2.689690] io scheduler bfq registered [ 2.691166] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.693414] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.696201] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.698544] ACPI: Power Button [PWRF] [ 2.701913] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.706547] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.714958] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.743728] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.769256] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.772415] Non-volatile memory driver v1.3 [ 2.773395] Linux agpgart interface v0.103 [ 2.799803] virtio_blk virtio1: [vda] 136552 512-byte logical blocks (69.9 MB/66.7 MiB) [ 2.802792] vda: detected capacity change from 0 to 69914624 [ 2.817367] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.820086] vdb: detected capacity change from 0 to 1073741824 [ 2.831179] libphy: Fixed MDIO Bus: probed [ 2.836508] usbcore: registered new interface driver usbserial_generic [ 2.839057] usbserial: USB Serial support registered for generic [ 2.841587] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.846082] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.848062] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.850942] mousedev: PS/2 mouse device common for all mice [ 2.854608] rtc_cmos 00:05: RTC can wake from S4 [ 2.858218] rtc_cmos 00:05: registered as rtc0 [ 2.859720] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.862702] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.866476] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.866599] intel_pstate: CPU model not supported [ 2.871382] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.873350] hid: raw HID events driver (C) Jiri Kosina [ 2.876088] usbcore: registered new interface driver usbhid [ 2.877332] usbhid: USB HID core driver [ 2.878264] drop_monitor: Initializing network drop monitor service [ 2.879644] Initializing XFRM netlink socket [ 2.880820] NET: Registered protocol family 10 [ 2.882971] Segment Routing with IPv6 [ 2.883774] NET: Registered protocol family 17 [ 2.885110] mpls_gso: MPLS GSO support [ 2.889772] RAS: Correctable Errors collector initialized. [ 2.891990] AVX version of gcm_enc/dec engaged. [ 2.893297] AES CTR mode by8 optimization enabled [ 2.951056] sched_clock: Marking stable (2951039680, 0)->(3879738675, -928698995) [ 2.953243] registered taskstats version 1 [ 2.954389] Loading compiled-in X.509 certificates [ 2.955821] zswap: loaded using pool lzo/zbud [ 2.973034] Key type big_key registered [ 2.985994] Key type encrypted registered [ 2.987617] ima: No TPM chip found, activating TPM-bypass! [ 2.989376] ima: Allocated hash algorithm: sha1 [ 2.991361] ima: No architecture policies found [ 2.993116] evm: Initialising EVM extended attributes: [ 2.994694] evm: security.selinux [ 2.995888] evm: security.ima [ 2.996578] evm: security.capability [ 2.997442] evm: HMAC attrs: 0x1 [ 2.998933] rtc_cmos 00:05: setting system clock to 2026-06-12 00:30:40 UTC (1781224240) [ 3.003072] debug: unmapping init [mem 0xffffffffa1003000-0xffffffffa11fffff] [ 3.004835] debug: unmapping init [mem 0xffffffff9fd82000-0xffffffffa0058fff] [ 3.013153] Write protecting the kernel read-only data: 28672k [ 3.015604] debug: unmapping init [mem 0xffffffff9e403000-0xffffffff9e5fffff] [ 3.017850] debug: unmapping init [mem 0xffffffff9ed14000-0xffffffff9edfffff] [ 3.046305] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.052727] systemd[1]: Detected virtualization kvm. [ 3.054248] systemd[1]: Detected architecture x86-64. [ 3.055561] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.081523] systemd[1]: No hostname configured. [ 3.082700] systemd[1]: Set hostname to . [ 3.084291] random: systemd: uninitialized urandom read (16 bytes read) [ 3.086145] systemd[1]: Initializing machine ID from random generator. [ 3.227281] random: systemd: uninitialized urandom read (16 bytes read) [ 3.229743] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.233509] random: systemd: uninitialized urandom read (16 bytes read) [ 3.235621] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.239982] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. Starting Journal Service... Starting Apply Kernel Variables... Starting Setup Virtual Console... [ OK ] Reached target Sockets. Starting Create Volatile Files and Directories... [ OK ] Reached target Swap. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.817269] device-mapper: uevent: version 1.0.3 [ 3.819156] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.532478] virtio_net virtio0 ens2: renamed from eth0 [ 4.550775] random: fast init done [ 4.588021] scsi host0: ata_piix [ 4.618730] scsi host1: ata_piix [ 4.620107] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.622234] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.942594] dracut-initqueue[592]: RTNETLINK answers: File exists [ 9.502493] random: crng init done [ 9.503889] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 9.902226] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.050951] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.324878] SELinux: Disabled at runtime. [ 11.388692] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.397422] systemd[1]: Detected virtualization kvm. [ 11.399209] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.917116] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.921204] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.926390] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.931596] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.935288] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.943161] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.947466] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-getty.slice. Starting Remount Root and Kernel File Systems... Mounting Huge Pages File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target Paths. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Activating swap /dev/disk/by-label/SWAP... Starting Create list of required st…ce nodes for the current kernel... Mounting POSIX Message Queue File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ 12.039168] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target RPC Port Mapper. Mounting Kernel Debug File System... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Starting Apply Kernel Variables... [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 12.347566] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.616739] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.617640] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.746117] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.761699] EDAC sbridge: Ver: 1.1.2 [ 13.969440] Key type dns_resolver registered [ 14.267710] NFS: Registering the id_resolver key type [ 14.269471] Key type id_resolver registered [ 14.271137] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg241-client login: [ 79.440728] libcfs: loading out-of-tree module taints kernel. [ 79.543692] Key type ._llcrypt registered [ 79.545248] Key type .llcrypt registered [ 80.316821] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 80.347173] alg: No test for adler32 (adler32-zlib) [ 81.833341] hrtimer: interrupt took 4866135 ns [ 82.326881] Lustre: Lustre: Build Version: 2.17.53_70_g68e5078 [ 83.392493] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 85.248205] Key type lgssc registered [ 87.059826] Lustre: Echo OBD driver; http://www.lustre.org/ [ 222.099292] Lustre: Mounted lustre-client [ 227.544302] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 247.331085] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing check_logdir /tmp/testlogs/ [ 247.776415] Lustre: lustre-OST0000-osc-ffff8a269035c800: disconnect after 23s idle [ 252.907207] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing yml_node [ 257.671385] Lustre: DEBUG MARKER: Client: 2.17.53.70 [ 260.250417] Lustre: DEBUG MARKER: MDS: 2.17.53.70 [ 263.059611] Lustre: DEBUG MARKER: OSS: 2.17.53.70 [ 265.020551] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Thu Jun 11 20:35:00 EDT 2026 [ 285.247097] Lustre: Unmounted lustre-client [ 317.169527] Key type lgssc unregistered [ 317.684274] LNet: 8400:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 317.700655] LNetError: 8400:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 317.729874] LNet: Removed LNI 192.168.202.41@tcp [ 318.949343] Key type .llcrypt unregistered [ 318.952697] Key type ._llcrypt unregistered [ 350.816978] Key type ._llcrypt registered [ 350.820348] Key type .llcrypt registered [ 351.046394] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 351.075353] alg: No test for adler32 (adler32-zlib) [ 352.091144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 352.160556] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 352.181777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 356.996275] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 357.145735] LNet: 9279:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 357.152259] LNetError: 9279:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 357.172476] LNet: Removed LNI 192.168.202.41@tcp [ 357.700385] Key type .llcrypt unregistered [ 357.702741] Key type ._llcrypt unregistered [ 367.079278] Key type ._llcrypt registered [ 367.085648] Key type .llcrypt registered [ 369.334023] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 20:36:44 (1781224604) [ 369.825888] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 369.850128] alg: No test for adler32 (adler32-zlib) [ 370.907428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 371.036979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 371.075750] LNet: 9697:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 371.101687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 371.188332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 371.207641] LNet: 9704:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 381.837484] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 20:36:57 (1781224617) [ 382.114505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 382.165712] LNet: 10282:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 382.247217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 382.408455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 391.702924] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 20:37:07 (1781224627) [ 391.951328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 392.009679] LNet: 10870:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 392.053217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 392.163513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 400.409410] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 20:37:16 (1781224636) [ 400.676505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 400.702387] LNet: 11457:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 400.722814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 400.792860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 408.416711] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 20:37:24 (1781224644) [ 408.618900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 408.668230] LNet: 12040:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 408.715384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 408.832489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 409.322668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 417.195653] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 20:37:32 (1781224652) [ 417.357748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 417.442596] LNet: 12631:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 417.486118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 417.571781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 418.357410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 418.992981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 428.170817] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 20:37:44 (1781224664) [ 428.505100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 428.558044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 428.667374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 436.713675] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 20:37:53 (1781224673) [ 437.058287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 437.089045] LNet: 13816:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 437.097146] LNet: 13816:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 437.149640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 437.240694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 437.276594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 437.929404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 438.010706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 438.453655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 438.517150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 439.293348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 439.335507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 440.038609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 440.072833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 440.798039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 440.862182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 441.331931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 441.371642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 441.694931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 441.718295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 441.944645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 441.966985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 442.188648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 442.208684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 442.741437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 442.778684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 443.221255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 443.271351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 443.925586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 443.999507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 452.512981] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 20:38:08 (1781224688) [ 452.924752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 453.028308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 453.158355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 453.209972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 460.980329] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 20:38:17 (1781224697) [ 461.226200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 461.258242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 461.342952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 461.377380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 469.714705] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 20:38:25 (1781224705) [ 469.824442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 469.887155] LNet: 15665:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 469.899706] LNet: 15665:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 469.937993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 470.071991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 470.122392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 479.123844] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 20:38:35 (1781224715) [ 479.364424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 479.440757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 479.604724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 479.641893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 488.088476] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 20:38:43 (1781224723) [ 488.265107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 488.329408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 488.410803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 488.449375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 496.200382] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 20:38:51 (1781224731) [ 496.307421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 496.353459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 496.472436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 496.527490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 504.184911] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 20:39:00 (1781224740) [ 504.590481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 504.668671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 504.891070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 505.638194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 513.682525] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 20:39:09 (1781224749) [ 513.789506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 513.866648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 514.103336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 514.504875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 523.373366] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 20:39:18 (1781224758) [ 523.555325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 523.694419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 523.885673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 524.144102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 532.838600] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 20:39:28 (1781224768) [ 532.957396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 533.017105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 533.543156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 533.920223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 543.040927] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 20:39:38 (1781224778) [ 543.230420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 543.263607] LNet: 20498:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 543.266775] LNet: 20498:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 543.301954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 543.440632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 543.495471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 551.692029] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 20:39:47 (1781224787) [ 551.886591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 551.958392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 552.050809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 552.105531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 560.846912] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 20:39:56 (1781224796) [ 561.009779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 561.134927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 561.263727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 568.992846] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 20:40:04 (1781224804) [ 569.231572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 569.308152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 569.488894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 577.519839] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 20:40:13 (1781224813) [ 577.628361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 577.692841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 577.817904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 578.161940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 586.162505] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 20:40:22 (1781224822) [ 586.558583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 586.586291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 586.653706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 586.697187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 594.351904] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 20:40:30 (1781224830) [ 594.729875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 594.774259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 594.857826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 594.908981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 603.019835] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 20:40:39 (1781224839) [ 603.146886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 603.215382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 603.426034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 603.504418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 612.231703] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 20:40:47 (1781224847) [ 612.503350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 612.537651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 612.640365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 612.674734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 621.077313] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 20:40:56 (1781224856) [ 621.428741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 621.499366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 621.584162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 621.704411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 621.723311] LNetError: 25818:0:(peer.c:2175:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 621.759229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 621.792269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 629.533872] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 20:41:05 (1781224865) [ 629.871059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 629.960963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 630.102060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.ggBbMiTT71 [ 630.231474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.ggBbMiTT71 [ 630.289774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.ggBbMiTT71 [ 638.885949] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 20:41:14 (1781224874) [ 639.007679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 639.050082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 639.112482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 639.140956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 639.185341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 647.096226] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 20:41:22 (1781224882) [ 647.278311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 647.324838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 647.418844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 651.352225] LNet: Message forwarding will be enabled in 2s [ 654.432157] LNet: Message forwarding enabled [ 654.924651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 654.940526] LNet: Message forwarding will stop in 2s to 6s [ 658.546366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 659.647775] LNet: Message forwarding disabled [ 661.640122] LNet: Message forwarding will be enabled in 2s [ 664.805136] LNet: Message forwarding enabled [ 669.425963] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 20:41:45 (1781224905) [ 669.529295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 670.925593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 671.042088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 678.462428] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 20:41:53 (1781224913) [ 678.622296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 678.664782] LNet: 28833:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 678.676112] LNet: 28833:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 13 previous similar messages [ 678.709852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 678.778202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 678.822823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 678.880932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 678.944523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 678.991866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 679.033963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 679.063293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 679.108810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 679.152335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 679.199807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 679.260498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 679.321110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 679.386599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 679.421604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 679.461343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 679.502943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 679.542526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 679.593505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 679.651257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 679.682546] LNetError: 28880:0:(router.c:624:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 687.807454] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 20:42:03 (1781224923) [ 687.947276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 688.016858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 688.263334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 696.739455] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 20:42:12 (1781224932) [ 699.210933] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg241-server [ 701.491699] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 20:42:17 (1781224937) [ 701.711085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 701.793216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 701.904530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 701.945486] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 703.563481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.202.[42-46]@tcp [ 703.586101] LNet: 30599:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 703.628910] LNetError: 30388:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.42@tcp is being used as a gateway but routing feature is not turned on [ 703.662934] LNetError: 30388:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp1 through 192.168.202.42@tcp has gone from up to down [ 704.685731] LNetError: 30388:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.42@tcp is being used as a gateway but routing feature is not turned on [ 704.706051] LNetError: 30388:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 705.708843] LNetError: 30388:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.43@tcp is being used as a gateway but routing feature is not turned on [ 705.721846] LNetError: 30388:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 707.752436] LNetError: 30388:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.43@tcp is being used as a gateway but routing feature is not turned on [ 707.775205] LNetError: 30388:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 710.861273] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 20:42:27 (1781224947) [ 711.848976] LNetError: 30388:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.42@tcp is being used as a gateway but routing feature is not turned on [ 711.857214] LNetError: 30388:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 713.456805] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg241-server [ 715.765884] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 20:42:31 (1781224951) [ 715.975672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 716.042264] LNetError: 31531:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 716.088278] LNet: Removed LNI 192.168.202.41@tcp [ 716.205703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 716.287711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 716.320569] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 717.545796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.202.[42-47/2]@tcp [ 717.580380] LNet: 31652:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 717.591762] LNet: 31652:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 717.629210] LNetError: 31539:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp103 through 192.168.202.44@tcp has gone from up to down [ 717.646871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.202.[42-47/2]@tcp [ 717.647584] LNetError: 31539:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 728.436584] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 20:42:43 (1781224963) [ 728.614029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 728.661764] LNetError: 32238:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 729.703150] LNet: Removed LNI 192.168.202.41@tcp [ 729.747620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 729.954811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 730.051938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 730.098639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 730.291921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 730.368777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 730.459701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 730.541491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 730.620823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 730.673517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 730.758224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 730.815587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 730.839723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 730.892358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 731.009706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 731.032587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 731.108530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 731.221673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 731.254706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 731.337139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 731.473881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 731.538404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 731.612860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 731.717794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 731.762729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 731.867918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 731.976904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 732.096131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 732.257564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 732.436785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 732.529128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 732.690885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 732.772797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 732.837338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 732.960118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 741.849700] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 20:42:57 (1781224977) [ 744.222742] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg241-server [ 746.822582] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 20:43:02 (1781224982) [ 749.749946] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg241-server [ 752.136996] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 20:43:07 (1781224987) [ 753.208471] Key type .llcrypt unregistered [ 753.215416] Key type ._llcrypt unregistered [ 761.375972] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 761.482833] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 761.501535] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 761.510980] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 764.530936] Key type ._llcrypt registered [ 764.533169] Key type .llcrypt registered [ 764.688789] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 764.703700] alg: No test for adler32 (adler32-zlib) [ 765.804943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 765.934195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 766.005204] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 766.935548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 766.969353] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 769.449514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 769.479768] LNet: Removed LNI 10.1.2.3@tcp [ 769.663076] LNet: 34373:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 769.672287] LNetError: 34373:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 769.707853] LNet: Removed LNI 192.168.202.41@tcp [ 770.707649] Key type .llcrypt unregistered [ 770.709951] Key type ._llcrypt unregistered [ 786.619739] Key type ._llcrypt registered [ 786.621276] Key type .llcrypt registered [ 788.467863] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 20:43:43 (1781225023) [ 788.781556] Key type .llcrypt unregistered [ 788.784703] Key type ._llcrypt unregistered [ 795.565726] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 795.806166] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 795.813639] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 795.829498] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 798.651945] Key type ._llcrypt registered [ 798.657390] Key type .llcrypt registered [ 799.000393] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 799.016686] alg: No test for adler32 (adler32-zlib) [ 799.867480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 799.929421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 799.949261] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 800.782663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 800.815938] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 805.063640] LNet: 35867:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 805.068586] LNetError: 35867:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 805.075571] LNet: Removed LNI 192.168.202.41@tcp [ 805.089793] LNet: Removed LNI 10.1.2.3@tcp [ 806.025427] Key type .llcrypt unregistered [ 806.027145] Key type ._llcrypt unregistered [ 823.541675] Key type ._llcrypt registered [ 823.544692] Key type .llcrypt registered [ 825.352474] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 20:44:20 (1781225060) [ 825.929357] Key type .llcrypt unregistered [ 825.932112] Key type ._llcrypt unregistered [ 834.332937] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 834.552634] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 834.565294] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 834.579405] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 837.355308] Key type ._llcrypt registered [ 837.358988] Key type .llcrypt registered [ 837.605622] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 837.648852] alg: No test for adler32 (adler32-zlib) [ 838.762193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 838.865261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 838.916735] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 841.815128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 841.852961] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 844.696792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 844.748132] LNet: Removed LNI 10.1.2.3@tcp [ 844.779557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 844.823624] LNet: Removed LNI 10.1.2.31@tcp [ 844.836263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 844.862970] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 845.857986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 845.873773] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 846.922497] LNet: 37782:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 846.943192] LNetError: 37782:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 846.946604] LNet: Removed LNI 10.1.2.3@tcp [ 846.957866] LNet: Removed LNI 10.1.2.31@tcp [ 847.592499] Key type .llcrypt unregistered [ 847.594870] Key type ._llcrypt unregistered [ 862.819630] Key type ._llcrypt registered [ 862.832715] Key type .llcrypt registered [ 864.695487] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 20:45:00 (1781225100) [ 865.117359] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 865.132222] alg: No test for adler32 (adler32-zlib) [ 866.179864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 866.338354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 866.391269] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 867.391126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 867.422331] LNet: 38735:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 867.434518] LNetError: 38735:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 867.453743] LNet: Removed LNI 192.168.202.41@tcp [ 867.487518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 867.578308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.202.41@tcp [ 867.616893] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 868.637649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.202.41@tcp [ 868.733262] LNet: Removed LNI 192.168.202.41@tcp [ 868.876289] LNet: 38870:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 869.793469] Key type .llcrypt unregistered [ 869.796286] Key type ._llcrypt unregistered [ 884.660628] Key type ._llcrypt registered [ 884.664714] Key type .llcrypt registered [ 886.160381] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 20:45:21 (1781225121) [ 888.434891] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg241-server [ 890.022662] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 20:45:26 (1781225126) [ 891.484395] Key type .llcrypt unregistered [ 891.485875] Key type ._llcrypt unregistered [ 899.218541] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 899.519940] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 899.530975] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 899.549946] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 902.396886] Key type ._llcrypt registered [ 902.400600] Key type .llcrypt registered [ 902.554682] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 902.563647] alg: No test for adler32 (adler32-zlib) [ 903.549144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 903.632382] LNet: Added LNI 192.168.202.41@tcp1 [8/256/0/180] [ 904.794658] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 907.651500] LNet: 40709:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 907.672804] LNetError: 40709:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 907.682538] LNet: Removed LNI 192.168.202.41@tcp1 [ 907.713226] LNet: Removed LNI 10.1.2.3@tcp2 [ 908.501642] Key type .llcrypt unregistered [ 908.508416] Key type ._llcrypt unregistered [ 922.273852] Key type ._llcrypt registered [ 922.276108] Key type .llcrypt registered [ 924.224564] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 20:45:59 (1781225159) [ 924.836407] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 924.853302] alg: No test for adler32 (adler32-zlib) [ 925.819350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 925.870356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 925.894931] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 926.655950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 926.675534] LNet: 41678:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 926.681644] LNetError: 41678:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 926.698788] LNet: Removed LNI 192.168.202.41@tcp [ 926.713634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 926.866528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 926.914745] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 935.505522] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 20:46:11 (1781225171) [ 935.711353] LNet: 42264:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 935.722821] LNetError: 42264:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 935.745777] LNet: Removed LNI 192.168.202.41@tcp [ 936.660663] Key type .llcrypt unregistered [ 936.669996] Key type ._llcrypt unregistered [ 945.112372] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 945.329056] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 945.333926] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 945.338925] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 948.350142] Key type ._llcrypt registered [ 948.352507] Key type .llcrypt registered [ 948.716300] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 948.759842] alg: No test for adler32 (adler32-zlib) [ 949.795371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 949.880775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 949.920500] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 950.841431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 950.881544] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 953.908084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 953.947731] LNet: 43007:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 953.956363] LNetError: 43007:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 953.963682] LNet: Removed LNI 192.168.202.41@tcp [ 953.981669] LNet: Removed LNI 10.1.2.3@tcp [ 954.018759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 954.102479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 954.137687] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 954.148401] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 954.858228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 954.905890] LNet: 43034:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 954.924739] LNetError: 43034:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 954.932395] LNetError: 43034:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 954.934702] LNet: Removed LNI 192.168.202.41@tcp [ 954.947068] LNet: Removed LNI 10.1.2.3@tcp [ 954.985649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 955.115078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 955.161254] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 955.170298] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 955.812931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 955.853407] LNet: Added LNI 192.168.202.41@tcp2 [8/256/0/180] [ 957.264894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 957.318456] LNet: 43197:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 957.326205] LNetError: 43197:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 957.331550] LNetError: 43197:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 957.337953] LNet: Removed LNI 192.168.202.41@tcp [ 957.360700] LNet: Removed LNI 10.1.2.3@tcp [ 957.376172] LNet: Removed LNI 192.168.202.41@tcp2 [ 957.399727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 957.473105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 957.509854] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 957.514471] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 957.524947] LNet: Added LNI 192.168.202.41@tcp2 [8/256/0/180] [ 967.975217] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 20:46:43 (1781225203) [ 968.161267] LNet: 43787:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 968.171647] LNetError: 43787:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 968.179099] LNetError: 43787:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 968.189773] LNet: Removed LNI 192.168.202.41@tcp [ 968.192368] LNet: Removed LNI 10.1.2.3@tcp [ 968.232328] LNet: Removed LNI 192.168.202.41@tcp2 [ 968.900885] Key type .llcrypt unregistered [ 968.903464] Key type ._llcrypt unregistered [ 976.904766] Key type ._llcrypt registered [ 976.906519] Key type .llcrypt registered [ 977.203886] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 977.231847] alg: No test for adler32 (adler32-zlib) [ 978.269112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 978.377334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 978.411344] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 979.602807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 979.652241] LNet: 44189:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 979.665399] LNetError: 44189:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 979.701266] LNet: Removed LNI 192.168.202.41@tcp [ 979.753141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 979.907670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 979.975865] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 980.891538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 980.929583] LNet: 44219:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 980.942737] LNetError: 44219:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 980.967807] LNet: Removed LNI 192.168.202.41@tcp [ 980.995222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 981.091123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 981.135163] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 982.036312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 982.067288] LNet: 44249:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 982.074138] LNetError: 44249:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 982.108282] LNet: Removed LNI 192.168.202.41@tcp [ 982.151493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 982.248887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 982.302349] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 983.269397] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 983.550315] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 983.556177] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 983.564039] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 986.305334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 986.362833] LNet: 44328:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 986.378540] LNetError: 44328:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 986.401630] LNet: Removed LNI 192.168.202.41@tcp [ 986.429674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 986.531333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 986.574649] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 995.873886] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 20:47:11 (1781225231) [ 996.002230] LNet: 44917:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 996.021701] LNetError: 44917:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 996.058738] LNet: Removed LNI 192.168.202.41@tcp [ 996.987159] Key type .llcrypt unregistered [ 996.991850] Key type ._llcrypt unregistered [ 1005.946036] Key type ._llcrypt registered [ 1005.948532] Key type .llcrypt registered [ 1006.433697] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1006.481160] alg: No test for adler32 (adler32-zlib) [ 1007.469578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1007.524912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1007.577990] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 1008.585548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1008.611522] LNet: 45324:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1008.620180] LNetError: 45324:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1008.640839] LNet: Removed LNI 192.168.202.41@tcp [ 1008.688831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1008.771265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 1008.840659] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 1016.386505] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 20:47:32 (1781225252) [ 1016.723379] LNet: 45918:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1016.730698] LNetError: 45918:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1016.752857] LNet: Removed LNI 192.168.202.41@tcp [ 1017.681572] Key type .llcrypt unregistered [ 1017.696265] Key type ._llcrypt unregistered [ 1025.253677] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1025.592412] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1025.604516] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1025.611613] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1028.439604] Key type ._llcrypt registered [ 1028.442823] Key type .llcrypt registered [ 1028.838733] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1028.880701] alg: No test for adler32 (adler32-zlib) [ 1029.842921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1029.934320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1029.975155] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 1030.900087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1030.943895] LNet: 46369:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1030.952445] LNetError: 46369:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1030.965383] LNet: Removed LNI 192.168.202.41@tcp [ 1031.009542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1031.097279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 1031.139812] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 1038.861901] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 20:47:54 (1781225274) [ 1039.109708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1039.163578] LNet: 46967:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1039.171605] LNetError: 46967:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1039.199028] LNet: Removed LNI 192.168.202.41@tcp [ 1039.270950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1039.408420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 1047.024763] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 20:48:02 (1781225282) [ 1047.173647] LNet: 47543:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1047.877730] Key type .llcrypt unregistered [ 1047.881480] Key type ._llcrypt unregistered [ 1055.112409] Key type ._llcrypt registered [ 1055.118986] Key type .llcrypt registered [ 1055.440454] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1055.468117] alg: No test for adler32 (adler32-zlib) [ 1056.391728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1056.445365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1056.466915] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 1057.614803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1057.635689] LNet: 47949:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1057.643984] LNetError: 47949:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1057.656937] LNet: Removed LNI 192.168.202.41@tcp [ 1057.678788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1057.736291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 1057.774588] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 1065.912554] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 20:48:21 (1781225301) [ 1066.033309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1066.071150] LNet: 48536:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1066.076467] LNetError: 48536:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1066.104026] LNet: Removed LNI 192.168.202.41@tcp [ 1066.154302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1066.280157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 1073.542742] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 20:48:29 (1781225309) [ 1075.353596] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 1077.212943] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 20:48:33 (1781225313) [ 1078.790849] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 1080.916495] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 20:48:36 (1781225316) [ 1082.856970] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 1084.865135] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 20:48:40 (1781225320) [ 1087.005469] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1089.321787] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 20:48:44 (1781225324) [ 1091.109454] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1093.161554] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 20:48:49 (1781225329) [ 1093.277946] LNet: 50587:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1093.960709] Key type .llcrypt unregistered [ 1093.963617] Key type ._llcrypt unregistered [ 1101.876797] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1102.278692] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1102.282634] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1102.285671] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1104.921445] Key type ._llcrypt registered [ 1104.923491] Key type .llcrypt registered [ 1105.207799] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1105.235445] alg: No test for adler32 (adler32-zlib) [ 1106.184171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1106.281860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1106.316749] LNet: Added LNI 192.168.202.41@tcp [8/512/0/180] [ 1106.332820] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1106.336656] LNet: 50917:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1106.483978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1106.521772] LNet: 50936:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1106.526822] LNetError: 50936:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1106.540173] LNet: Removed LNI 192.168.202.41@tcp [ 1106.541796] LNetError: 50936:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1106.572516] LNet: Removed LNI 10.1.2.3@tcp [ 1106.620756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1106.723841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1106.775210] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 1106.834635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1106.855428] LNet: 50960:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1106.860027] LNetError: 50960:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1106.882991] LNet: Removed LNI 192.168.202.41@tcp [ 1106.932949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1106.992796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1107.027359] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 1107.032705] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1107.035951] LNet: 50968:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1107.187764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1107.221177] LNet: 50986:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1107.225811] LNetError: 50986:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1107.230545] LNet: Removed LNI 192.168.202.41@tcp [ 1107.233170] LNetError: 50986:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1107.242655] LNet: Removed LNI 10.1.2.3@tcp [ 1107.264410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1107.329560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1107.348796] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 1107.355155] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1107.358559] LNet: 50994:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1107.536253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1107.549268] LNet: 51020:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1107.552542] LNetError: 51020:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1107.554747] LNet: Removed LNI 192.168.202.41@tcp [ 1107.557978] LNetError: 51020:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1107.573467] LNet: Removed LNI 10.1.2.3@tcp [ 1107.590496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1107.681570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1107.702562] LNet: Added LNI 192.168.202.41@tcp [8/256/0/182] [ 1107.714921] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1107.867149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1107.932684] LNet: 51048:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1107.940927] LNetError: 51048:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1107.952276] LNet: Removed LNI 192.168.202.41@tcp [ 1107.955594] LNetError: 51048:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1107.976036] LNet: Removed LNI 10.1.2.3@tcp [ 1108.042809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1108.125975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1108.154633] LNet: Added LNI 192.168.202.41@tcp [8/256/0/182] [ 1108.161158] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1108.163140] LNet: 51056:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1108.259826] LNet: Removed LNI 192.168.202.41@tcp [ 1108.269928] LNet: Removed LNI 10.1.2.3@tcp [ 1108.923609] Key type .llcrypt unregistered [ 1108.926396] Key type ._llcrypt unregistered [ 1122.312453] Key type ._llcrypt registered [ 1122.315349] Key type .llcrypt registered [ 1123.967938] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 20:49:19 (1781225359) [ 1124.495701] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1124.518174] alg: No test for adler32 (adler32-zlib) [ 1125.500499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1125.543671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1125.565289] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 1126.471904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.202.41 [ 1126.505546] LNet: 52032:0:(config.c:1627:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1126.516132] LNet: Removed LNI 192.168.202.41@tcp [ 1133.420855] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 20:49:29 (1781225369) [ 1133.706218] LNet: 52602:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1133.716893] LNetError: 52602:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1133.733792] LNet: Removed LNI 192.168.202.41@tcp [ 1134.306411] Key type .llcrypt unregistered [ 1134.308045] Key type ._llcrypt unregistered [ 1140.985803] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1141.243316] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1144.013857] Key type ._llcrypt registered [ 1144.016644] Key type .llcrypt registered [ 1144.174307] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1144.184762] alg: No test for adler32 (adler32-zlib) [ 1145.661585] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1153.790369] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 20:49:49 (1781225389) [ 1153.989354] LNet: 53508:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1153.999100] LNetError: 53508:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1154.041160] LNet: Removed LNI 10.1.2.3@tcp [ 1154.772371] Key type .llcrypt unregistered [ 1154.775673] Key type ._llcrypt unregistered [ 1162.730892] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1163.059683] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1163.062455] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1163.067361] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1166.095383] Key type ._llcrypt registered [ 1166.100255] Key type .llcrypt registered [ 1166.434534] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1166.454925] alg: No test for adler32 (adler32-zlib) [ 1167.470370] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1175.491655] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 20:50:11 (1781225411) [ 1177.138150] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1179.201496] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 20:50:15 (1781225415) [ 1179.662371] LNet: 54700:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1179.674567] LNetError: 54700:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1179.700214] LNet: Removed LNI 10.1.2.3@tcp [ 1180.416498] Key type .llcrypt unregistered [ 1180.418071] Key type ._llcrypt unregistered [ 1189.252386] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1189.487335] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1189.491274] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1189.495895] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1192.337764] Key type ._llcrypt registered [ 1192.340292] Key type .llcrypt registered [ 1192.631442] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1192.668164] alg: No test for adler32 (adler32-zlib) [ 1193.662232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1193.744392] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1205.244826] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 20:50:41 (1781225441) [ 1206.084171] LNet: 55930:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1206.098753] LNetError: 55930:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1206.122701] LNet: Removed LNI 10.1.2.3@tcp [ 1206.989638] Key type .llcrypt unregistered [ 1206.992811] Key type ._llcrypt unregistered [ 1216.005231] Key type ._llcrypt registered [ 1216.008826] Key type .llcrypt registered [ 1216.334946] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1216.360880] alg: No test for adler32 (adler32-zlib) [ 1217.341096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1217.403596] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 1226.251909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.141@tcp [ 1227.135541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.41@tcp -d 192.168.202.141@tcp -m GET -e local_interrupt -r 1 [ 1227.188277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.41@tcp -d 192.168.202.41@tcp -m GET -e local_interrupt -r 1 [ 1227.236314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.141@tcp [ 1227.253554] LNet: There was an unexpected network error while writing to 192.168.202.141: rc = -22 [ 1227.575786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1227.626697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1227.667689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1227.687541] LNet: 1 local NIs in recovery (showing 1): 192.168.202.41@tcp [ 1227.751770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1227.824483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1227.894291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1227.946842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1227.999362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1228.023974] LNet: 56368:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1228.074908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1228.154834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1228.233211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1228.262041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1228.300723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1228.344930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1228.397577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1229.118234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.41@tcp -d 192.168.202.141@tcp -m GET -e local_dropped -r 1 [ 1229.159658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.41@tcp -d 192.168.202.41@tcp -m GET -e local_dropped -r 1 [ 1229.216274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.141@tcp [ 1229.236399] LNet: There was an unexpected network error while writing to 192.168.202.141: rc = -22 [ 1229.454534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1229.488413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1229.513106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1229.542464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1229.583417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1229.622880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1229.653708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1229.684623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1229.707444] LNet: 56477:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1229.723254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1229.754866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1229.803574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1229.864898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1229.915250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1229.968205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1230.003793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1230.657955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.41@tcp -d 192.168.202.141@tcp -m GET -e local_aborted -r 1 [ 1230.716906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.41@tcp -d 192.168.202.41@tcp -m GET -e local_aborted -r 1 [ 1230.814605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.141@tcp [ 1230.853882] LNet: There was an unexpected network error while writing to 192.168.202.141: rc = -22 [ 1231.220170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1231.265411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1231.335421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1231.378571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1231.448494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1231.494652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1231.572078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1231.630492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1231.655879] LNet: 56586:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1231.693277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1231.754975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1231.818639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1231.887716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1231.965898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1232.020170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1232.073866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1232.786743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.41@tcp -d 192.168.202.141@tcp -m GET -e local_no_route -r 1 [ 1232.819526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.41@tcp -d 192.168.202.41@tcp -m GET -e local_no_route -r 1 [ 1232.847460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.141@tcp [ 1232.863407] LNet: There was an unexpected network error while writing to 192.168.202.141: rc = -22 [ 1232.867868] LNet: Skipped 1 previous similar message [ 1233.061651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1233.087693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1233.115766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1233.149582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1233.179414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1233.206285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1233.236749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1233.277559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1233.313851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1233.345543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1233.370985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1233.396843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1233.419503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1233.444855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1233.471813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1234.103588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.41@tcp -d 192.168.202.141@tcp -m GET -e local_timeout -r 1 [ 1234.168197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.41@tcp -d 192.168.202.41@tcp -m GET -e local_timeout -r 1 [ 1234.222682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.141@tcp [ 1234.610170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1234.650702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1234.720831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1234.823147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1234.893561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1234.930845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1234.982523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1235.029327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1235.049417] LNet: 56805:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1235.057233] LNet: 56805:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1235.082803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1235.152950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1235.215730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1235.271617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1235.343589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1235.454429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1235.483682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1236.108914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.41@tcp -d 192.168.202.141@tcp -m GET -e local_error -r 1 [ 1236.198533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.41@tcp -d 192.168.202.41@tcp -m GET -e local_error -r 1 [ 1236.292724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.141@tcp [ 1236.661595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1236.685575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1236.712415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1236.735665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1236.776906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1236.810544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1236.848985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1236.890497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1236.934728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1236.975678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1237.016566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1237.059435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1237.109652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1237.142718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1237.200295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1237.683485] LNet: 56955:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1237.696242] LNetError: 56955:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1237.711392] LNet: Removed LNI 192.168.202.41@tcp [ 1238.190778] Key type .llcrypt unregistered [ 1238.195448] Key type ._llcrypt unregistered [ 1261.530447] Key type ._llcrypt registered [ 1261.535281] Key type .llcrypt registered [ 1263.214781] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 20:51:38 (1781225498) [ 1264.193531] Key type .llcrypt unregistered [ 1264.196123] Key type ._llcrypt unregistered [ 1270.883436] Key type ._llcrypt registered [ 1270.888933] Key type .llcrypt registered [ 1271.185947] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1271.218486] alg: No test for adler32 (adler32-zlib) [ 1272.189060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1272.212728] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 1281.647766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.141@tcp [ 1295.132313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1295.173797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1295.236040] LNet: Added LNI 192.168.202.41@tcp1 [8/256/0/180] [ 1297.007137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.41@tcp -d 192.168.202.141@tcp -m GET -e local_interrupt -r 1 [ 1297.061862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.41@tcp -d 192.168.202.141@tcp1 -m GET -e local_interrupt -r 1 [ 1297.109716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.41@tcp -d 192.168.202.41@tcp -m GET -e local_interrupt -r 1 [ 1297.152493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.41@tcp -d 192.168.202.41@tcp1 -m GET -e local_interrupt -r 1 [ 1297.195893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.41@tcp1 -d 192.168.202.141@tcp -m GET -e local_interrupt -r 1 [ 1297.230425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.41@tcp1 -d 192.168.202.141@tcp1 -m GET -e local_interrupt -r 1 [ 1297.277340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.41@tcp1 -d 192.168.202.41@tcp -m GET -e local_interrupt -r 1 [ 1297.353756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.41@tcp1 -d 192.168.202.41@tcp1 -m GET -e local_interrupt -r 1 [ 1297.432797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.141@tcp [ 1297.454669] LNet: There was an unexpected network error while writing to 192.168.202.141: rc = -22 [ 1297.462762] LNet: 1 local NIs in recovery (showing 1): 192.168.202.41@tcp [ 1297.770215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1297.817384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1297.859241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1297.890302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1297.934529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1297.980312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1298.012801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1298.050732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1298.068400] LNet: 58423:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1298.126228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1298.189860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1298.245124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1298.309837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1298.388466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1298.446963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1298.522860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1299.216629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.41@tcp -d 192.168.202.141@tcp -m GET -e local_dropped -r 1 [ 1299.293716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.41@tcp -d 192.168.202.141@tcp1 -m GET -e local_dropped -r 1 [ 1299.365641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.41@tcp -d 192.168.202.41@tcp -m GET -e local_dropped -r 1 [ 1299.426807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.41@tcp -d 192.168.202.41@tcp1 -m GET -e local_dropped -r 1 [ 1299.494441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.41@tcp1 -d 192.168.202.141@tcp -m GET -e local_dropped -r 1 [ 1299.555369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.41@tcp1 -d 192.168.202.141@tcp1 -m GET -e local_dropped -r 1 [ 1299.677537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.41@tcp1 -d 192.168.202.41@tcp -m GET -e local_dropped -r 1 [ 1299.746858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.41@tcp1 -d 192.168.202.41@tcp1 -m GET -e local_dropped -r 1 [ 1299.818792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.141@tcp [ 1299.845743] LNet: There was an unexpected network error while writing to 192.168.202.141: rc = -22 [ 1299.851632] LNet: Skipped 2 previous similar messages [ 1300.142319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1300.178186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1300.211935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1300.252996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1300.286344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1300.322821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1300.352588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1300.396472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1300.428416] LNet: 58545:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1300.465991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1300.513450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1300.549777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1300.591350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1300.644869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1300.696619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1300.740316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1301.540112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.41@tcp -d 192.168.202.141@tcp -m GET -e local_aborted -r 1 [ 1301.624376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.41@tcp -d 192.168.202.141@tcp1 -m GET -e local_aborted -r 1 [ 1301.681061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.41@tcp -d 192.168.202.41@tcp -m GET -e local_aborted -r 1 [ 1301.783867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.41@tcp -d 192.168.202.41@tcp1 -m GET -e local_aborted -r 1 [ 1301.847375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.41@tcp1 -d 192.168.202.141@tcp -m GET -e local_aborted -r 1 [ 1301.912367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.41@tcp1 -d 192.168.202.141@tcp1 -m GET -e local_aborted -r 1 [ 1301.959872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.41@tcp1 -d 192.168.202.41@tcp -m GET -e local_aborted -r 1 [ 1301.991666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.41@tcp1 -d 192.168.202.41@tcp1 -m GET -e local_aborted -r 1 [ 1302.042563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.141@tcp [ 1302.082115] LNet: There was an unexpected network error while writing to 192.168.202.141: rc = -22 [ 1302.087337] LNet: Skipped 2 previous similar messages [ 1302.321766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1302.345118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1302.380762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1302.418345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1302.470190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1302.509612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1302.546370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1302.601282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1302.628092] LNet: 58666:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1302.650600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1302.704824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1302.762983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1302.819299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1302.898678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1302.952265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1302.995968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1303.857529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.41@tcp -d 192.168.202.141@tcp -m GET -e local_no_route -r 1 [ 1303.943898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.41@tcp -d 192.168.202.141@tcp1 -m GET -e local_no_route -r 1 [ 1304.022282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.41@tcp -d 192.168.202.41@tcp -m GET -e local_no_route -r 1 [ 1304.075170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.41@tcp -d 192.168.202.41@tcp1 -m GET -e local_no_route -r 1 [ 1304.128847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.41@tcp1 -d 192.168.202.141@tcp -m GET -e local_no_route -r 1 [ 1304.194680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.41@tcp1 -d 192.168.202.141@tcp1 -m GET -e local_no_route -r 1 [ 1304.268717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.41@tcp1 -d 192.168.202.41@tcp -m GET -e local_no_route -r 1 [ 1304.341535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.41@tcp1 -d 192.168.202.41@tcp1 -m GET -e local_no_route -r 1 [ 1304.398181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.141@tcp [ 1304.452332] LNet: There was an unexpected network error while writing to 192.168.202.141: rc = -22 [ 1304.465828] LNet: Skipped 2 previous similar messages [ 1304.802685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1304.860904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1304.903350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1304.955877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1305.002529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1305.037730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1305.085637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1305.146872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1305.173175] LNet: 58787:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1305.191153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1305.226280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1305.261686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1305.299749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1305.342481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1305.408662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1305.468817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1306.235957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.41@tcp -d 192.168.202.141@tcp -m GET -e local_timeout -r 1 [ 1306.289255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.41@tcp -d 192.168.202.141@tcp1 -m GET -e local_timeout -r 1 [ 1306.352294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.41@tcp -d 192.168.202.41@tcp -m GET -e local_timeout -r 1 [ 1306.411593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.41@tcp -d 192.168.202.41@tcp1 -m GET -e local_timeout -r 1 [ 1306.480724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.41@tcp1 -d 192.168.202.141@tcp -m GET -e local_timeout -r 1 [ 1306.555152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.41@tcp1 -d 192.168.202.141@tcp1 -m GET -e local_timeout -r 1 [ 1306.602411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.41@tcp1 -d 192.168.202.41@tcp -m GET -e local_timeout -r 1 [ 1306.639839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.41@tcp1 -d 192.168.202.41@tcp1 -m GET -e local_timeout -r 1 [ 1306.673798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.141@tcp [ 1306.911894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1306.948596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1306.987197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1307.022095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1307.054173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1307.086039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1307.115269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1307.150872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1307.188953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1307.215402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1307.253877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1307.285647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1307.315897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1307.346383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1307.377649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1308.080545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.41@tcp -d 192.168.202.141@tcp -m GET -e local_error -r 1 [ 1308.146720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.41@tcp -d 192.168.202.141@tcp1 -m GET -e local_error -r 1 [ 1308.201089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.41@tcp -d 192.168.202.41@tcp -m GET -e local_error -r 1 [ 1308.272913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.41@tcp -d 192.168.202.41@tcp1 -m GET -e local_error -r 1 [ 1308.331722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.41@tcp1 -d 192.168.202.141@tcp -m GET -e local_error -r 1 [ 1308.395454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.41@tcp1 -d 192.168.202.141@tcp1 -m GET -e local_error -r 1 [ 1308.475695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.41@tcp1 -d 192.168.202.41@tcp -m GET -e local_error -r 1 [ 1308.537547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.41@tcp1 -d 192.168.202.41@tcp1 -m GET -e local_error -r 1 [ 1308.603621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.141@tcp [ 1308.632413] LNet: There was an unexpected network error while writing to 192.168.202.141: rc = -22 [ 1308.637385] LNet: Skipped 5 previous similar messages [ 1309.032919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1309.076133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1309.119398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1309.162932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1309.218737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1309.272137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1309.304289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1309.345885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1309.375304] LNet: 59030:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1309.384834] LNet: 59030:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1309.411248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1309.448917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1309.508226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1309.556382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1309.623992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1309.683322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1309.728676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1316.575850] LNet: 59106:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1316.581662] LNetError: 59106:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1316.588962] LNet: Removed LNI 192.168.202.41@tcp [ 1316.636449] LNet: Removed LNI 192.168.202.41@tcp1 [ 1317.196662] Key type .llcrypt unregistered [ 1317.198553] Key type ._llcrypt unregistered [ 1339.559590] Key type ._llcrypt registered [ 1339.561945] Key type .llcrypt registered [ 1341.392145] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 20:52:56 (1781225576) [ 1342.447401] Key type .llcrypt unregistered [ 1342.449668] Key type ._llcrypt unregistered [ 1350.808245] Key type ._llcrypt registered [ 1350.823280] Key type .llcrypt registered [ 1351.130564] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1351.161138] alg: No test for adler32 (adler32-zlib) [ 1352.259870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1352.302858] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 1362.253636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.141@tcp [ 1363.135877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.41@tcp -d 192.168.202.141@tcp -m GET -e remote_dropped -r 1 [ 1363.187729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.41@tcp -d 192.168.202.41@tcp -m GET -e remote_dropped -r 1 [ 1363.251990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.141@tcp [ 1363.290290] LNet: There was an unexpected network error while writing to 192.168.202.141: rc = -22 [ 1363.637526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1363.680378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1363.722482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1363.812663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1363.869023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1363.925291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1363.972220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1364.013221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1364.040167] LNet: 60356:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1364.064602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1364.172852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1364.224331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1364.265193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1364.308174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1364.375761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1364.429530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1365.221976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.41@tcp -d 192.168.202.141@tcp -m GET -e remote_error -r 1 [ 1365.279453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.41@tcp -d 192.168.202.41@tcp -m GET -e remote_error -r 1 [ 1365.343724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.141@tcp [ 1365.396170] LNet: There was an unexpected network error while writing to 192.168.202.141: rc = -22 [ 1365.740183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1365.762899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1365.783679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1365.812757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1365.845403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1365.869498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1365.897501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1365.928289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1365.947287] LNet: 60466:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1365.975431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1366.037624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1366.080714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1366.120507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1366.163457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1366.206411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1366.262462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1367.039802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.41@tcp -d 192.168.202.141@tcp -m GET -e remote_timeout -r 1 [ 1367.073017] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.41@tcp -d 192.168.202.41@tcp -m GET -e remote_timeout -r 1 [ 1367.129838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.141@tcp [ 1367.171088] LNet: There was an unexpected network error while writing to 192.168.202.141: rc = -22 [ 1367.533740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1367.583932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1367.661742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1367.754825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1367.823272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1367.876497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1367.915594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1367.972752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1367.998751] LNet: 60575:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1368.039435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1368.099060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1368.154501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1368.185609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1368.220992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1368.269955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1368.330382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1368.695655] LNet: 60616:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1368.701506] LNetError: 60616:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1368.722751] LNet: Removed LNI 192.168.202.41@tcp [ 1369.461874] Key type .llcrypt unregistered [ 1369.467290] Key type ._llcrypt unregistered [ 1393.794972] Key type ._llcrypt registered [ 1393.798403] Key type .llcrypt registered [ 1395.706335] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 20:53:51 (1781225631) [ 1397.032430] Key type .llcrypt unregistered [ 1397.034532] Key type ._llcrypt unregistered [ 1405.921359] Key type ._llcrypt registered [ 1405.926260] Key type .llcrypt registered [ 1406.270387] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1406.309882] alg: No test for adler32 (adler32-zlib) [ 1407.320132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1407.368649] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 1416.259747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.141@tcp [ 1428.519156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1428.563854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1428.608808] LNet: Added LNI 192.168.202.41@tcp1 [8/256/0/180] [ 1430.364730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.41@tcp -d 192.168.202.141@tcp -m GET -e remote_dropped -r 1 [ 1430.406642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.41@tcp -d 192.168.202.141@tcp1 -m GET -e remote_dropped -r 1 [ 1430.469809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.41@tcp -d 192.168.202.41@tcp -m GET -e remote_dropped -r 1 [ 1430.569363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.41@tcp -d 192.168.202.41@tcp1 -m GET -e remote_dropped -r 1 [ 1430.659132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.41@tcp1 -d 192.168.202.141@tcp -m GET -e remote_dropped -r 1 [ 1430.740179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.41@tcp1 -d 192.168.202.141@tcp1 -m GET -e remote_dropped -r 1 [ 1430.792724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.41@tcp1 -d 192.168.202.41@tcp -m GET -e remote_dropped -r 1 [ 1430.827665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.41@tcp1 -d 192.168.202.41@tcp1 -m GET -e remote_dropped -r 1 [ 1430.849426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.141@tcp [ 1430.864653] LNet: There was an unexpected network error while writing to 192.168.202.141: rc = -22 [ 1430.870748] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.141@tcp [ 1431.241194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1431.270510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1431.303410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1431.344906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1431.393545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1431.457832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1431.519446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1431.581824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1431.604376] LNet: 62087:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1431.626016] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1431.684573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1431.744761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1431.797968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1431.860589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1431.937811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1432.009326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1432.071646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1432.874602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.41@tcp -d 192.168.202.141@tcp -m GET -e remote_error -r 1 [ 1432.925363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.41@tcp -d 192.168.202.141@tcp1 -m GET -e remote_error -r 1 [ 1432.993090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.41@tcp -d 192.168.202.41@tcp -m GET -e remote_error -r 1 [ 1433.056056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.41@tcp -d 192.168.202.41@tcp1 -m GET -e remote_error -r 1 [ 1433.113599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.41@tcp1 -d 192.168.202.141@tcp -m GET -e remote_error -r 1 [ 1433.154932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.41@tcp1 -d 192.168.202.141@tcp1 -m GET -e remote_error -r 1 [ 1433.185632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.41@tcp1 -d 192.168.202.41@tcp -m GET -e remote_error -r 1 [ 1433.215734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.41@tcp1 -d 192.168.202.41@tcp1 -m GET -e remote_error -r 1 [ 1433.263296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.141@tcp [ 1433.297651] LNet: There was an unexpected network error while writing to 192.168.202.141: rc = -22 [ 1433.306796] LNet: Skipped 2 previous similar messages [ 1433.672867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1433.757725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1433.805491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1433.869470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1433.898854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1433.974844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1434.058416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1434.159407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1434.199699] LNet: 62211:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1434.237935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1434.317492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1434.370898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1434.422534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1434.492971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1434.560927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1434.624872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1434.743974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1435.537632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.41@tcp -d 192.168.202.141@tcp -m GET -e remote_timeout -r 1 [ 1435.585901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.41@tcp -d 192.168.202.141@tcp1 -m GET -e remote_timeout -r 1 [ 1435.662515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.41@tcp -d 192.168.202.41@tcp -m GET -e remote_timeout -r 1 [ 1435.751640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.41@tcp -d 192.168.202.41@tcp1 -m GET -e remote_timeout -r 1 [ 1435.813287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.41@tcp1 -d 192.168.202.141@tcp -m GET -e remote_timeout -r 1 [ 1435.860785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.41@tcp1 -d 192.168.202.141@tcp1 -m GET -e remote_timeout -r 1 [ 1435.928628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.41@tcp1 -d 192.168.202.41@tcp -m GET -e remote_timeout -r 1 [ 1435.991376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.41@tcp1 -d 192.168.202.41@tcp1 -m GET -e remote_timeout -r 1 [ 1436.065560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.141@tcp [ 1436.100485] LNet: There was an unexpected network error while writing to 192.168.202.141: rc = -22 [ 1436.429409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1436.489058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1436.545830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1436.600237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1436.671793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1436.746457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1436.826943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1436.870873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1436.903476] LNet: 62334:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1436.937622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1436.995097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1437.014830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1437.032547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1437.058725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1437.105031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1437.157873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1437.210161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1443.379315] LNet: 62412:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1443.389408] LNetError: 62412:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1443.403671] LNet: Removed LNI 192.168.202.41@tcp [ 1443.415404] LNet: Removed LNI 192.168.202.41@tcp1 [ 1443.907607] Key type .llcrypt unregistered [ 1443.909780] Key type ._llcrypt unregistered [ 1465.913070] Key type ._llcrypt registered [ 1465.915488] Key type .llcrypt registered [ 1467.691610] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 20:55:03 (1781225703) [ 1469.808093] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1471.771652] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 20:55:07 (1781225707) [ 1473.090257] Key type .llcrypt unregistered [ 1473.092160] Key type ._llcrypt unregistered [ 1481.630516] Key type ._llcrypt registered [ 1481.635920] Key type .llcrypt registered [ 1481.945872] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1481.986919] alg: No test for adler32 (adler32-zlib) [ 1482.995879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1483.025734] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 1491.527854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.141@tcp [ 1492.226477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.41@tcp -d 192.168.202.141@tcp -m GET -e network_timeout -r 1 [ 1492.305127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.41@tcp -d 192.168.202.41@tcp -m GET -e network_timeout -r 1 [ 1492.396973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.141@tcp [ 1492.448045] LNet: There was an unexpected network error while writing to 192.168.202.141: rc = -22 [ 1492.909542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1492.943613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1492.988165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1493.041600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1493.077867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1493.129934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1493.200489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1493.259269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1493.282736] LNet: 1 local NIs in recovery (showing 1): 192.168.202.41@tcp [ 1493.297417] LNet: 63957:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1493.298189] LNet: There was an unexpected network error while writing to 192.168.202.41: rc = -22 [ 1493.343424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1493.418391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1493.501718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1493.581693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1493.671298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1493.741446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1494.165348] LNet: 63996:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1494.195442] LNetError: 63996:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1495.208530] LNet: Removed LNI 192.168.202.41@tcp [ 1495.843586] Key type .llcrypt unregistered [ 1495.855867] Key type ._llcrypt unregistered [ 1519.255690] Key type ._llcrypt registered [ 1519.258218] Key type .llcrypt registered [ 1519.708904] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1519.735131] alg: No test for adler32 (adler32-zlib) [ 1520.701791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1520.726419] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 1529.974714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.141@tcp [ 1539.835755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1539.897343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1539.926933] LNet: Added LNI 192.168.202.41@tcp1 [8/256/0/180] [ 1542.114674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.41@tcp -d 192.168.202.141@tcp -m GET -e network_timeout -r 1 [ 1542.202389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.41@tcp -d 192.168.202.141@tcp1 -m GET -e network_timeout -r 1 [ 1542.296103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.41@tcp -d 192.168.202.41@tcp -m GET -e network_timeout -r 1 [ 1542.384710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.41@tcp -d 192.168.202.41@tcp1 -m GET -e network_timeout -r 1 [ 1542.475460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.41@tcp1 -d 192.168.202.141@tcp -m GET -e network_timeout -r 1 [ 1542.558320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.41@tcp1 -d 192.168.202.141@tcp1 -m GET -e network_timeout -r 1 [ 1542.606795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.41@tcp1 -d 192.168.202.41@tcp -m GET -e network_timeout -r 1 [ 1542.658548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.41@tcp1 -d 192.168.202.41@tcp1 -m GET -e network_timeout -r 1 [ 1542.687813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.141@tcp [ 1542.714586] LNet: There was an unexpected network error while writing to 192.168.202.141: rc = -22 [ 1542.955902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1543.022642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1543.074454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1543.103469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1543.140676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1543.197880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1543.252935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1543.265508] LNet: 1 local NIs in recovery (showing 1): 192.168.202.41@tcp [ 1543.266084] LNet: There was an unexpected network error while writing to 192.168.202.141: rc = -22 [ 1543.302544] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.141@tcp [ 1543.344144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1543.373430] LNet: 64876:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1543.391647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1543.445151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1543.489875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1543.536881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1543.593556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1543.654027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1549.925192] LNet: There was an unexpected network error while writing to 192.168.202.141: rc = -22 [ 1549.930837] LNet: Skipped 1 previous similar message [ 1550.301144] LNet: 64950:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1550.316383] LNetError: 64950:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1550.323316] LNet: Removed LNI 192.168.202.41@tcp [ 1550.333921] LNet: Removed LNI 192.168.202.41@tcp1 [ 1550.889705] Key type .llcrypt unregistered [ 1550.891298] Key type ._llcrypt unregistered [ 1571.407333] Key type ._llcrypt registered [ 1571.409637] Key type .llcrypt registered [ 1573.538773] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 20:56:48 (1781225808) [ 1573.875429] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1573.899695] alg: No test for adler32 (adler32-zlib) [ 1575.075451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1575.184353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1575.252560] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 1577.093935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1577.171648] LNet: Added LNI 192.168.202.41@tcp1 [8/256/0/180] [ 1579.105852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.41@tcp [ 1579.285665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1579.401455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1579.585758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1579.664490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1579.775061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1579.863532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.41@tcp [ 1579.919792] LNetError: 66054:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.202.41@tcp to recovery [ 1580.260591] LNet: 1 local NIs in recovery (showing 1): 192.168.202.41@tcp [ 1580.261263] LNet: There was an unexpected network error while writing to 192.168.202.41: rc = -22 [ 1582.306387] LNet: There was an unexpected network error while writing to 192.168.202.41: rc = -22 [ 1583.714521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1586.401270] LNet: There was an unexpected network error while writing to 192.168.202.41: rc = -22 [ 1587.608802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1587.671783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1587.817231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1587.868910] LNet: 66161:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1587.893261] LNetError: 66161:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1587.926711] LNet: Removed LNI 192.168.202.41@tcp [ 1587.938765] LNet: Removed LNI 192.168.202.41@tcp1 [ 1588.007401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1588.120480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1588.241426] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 1589.239170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1589.264655] LNet: Added LNI 192.168.202.41@tcp1 [8/256/0/180] [ 1590.716454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.41@tcp [ 1590.743490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1590.792234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1590.832893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1590.859888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1590.900420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1590.935305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1590.991727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.41@tcp [ 1591.018277] LNetError: 66433:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.202.41@tcp to recovery [ 1591.143382] LNet: There was an unexpected network error while writing to 192.168.202.41: rc = -22 [ 1593.760990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1597.281586] LNet: There was an unexpected network error while writing to 192.168.202.41: rc = -22 [ 1597.300059] LNet: Skipped 1 previous similar message [ 1602.494703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1602.547677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1602.600130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1612.357786] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 20:57:27 (1781225847) [ 1612.438730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1612.483066] LNet: 67133:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1612.494768] LNetError: 67133:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1612.518829] LNetError: 67133:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1612.526699] LNet: Removed LNI 192.168.202.41@tcp [ 1612.557618] LNet: Removed LNI 192.168.202.41@tcp1 [ 1612.623959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1612.767656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1612.793132] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 1613.970155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1614.004709] LNet: Added LNI 192.168.202.41@tcp1 [8/256/0/180] [ 1615.613301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.41@tcp [ 1615.747357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1615.825833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1615.902331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1615.947789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1615.987384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1616.047447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.41@tcp --health 0 [ 1616.186372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1616.803691] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.41@tcp [ 1617.825393] LNet: There was an unexpected network error while writing to 192.168.202.41: rc = -22 [ 1617.831408] LNet: Skipped 1 previous similar message [ 1632.405221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1632.724547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1632.783443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1632.866932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.41@tcp --health 500 [ 1633.077982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1638.949462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1638.995873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1639.012443] LNet: 67626:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1639.026053] LNetError: 67626:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1639.036133] LNetError: 67626:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1639.040411] LNet: Removed LNI 192.168.202.41@tcp [ 1639.052888] LNet: Removed LNI 192.168.202.41@tcp1 [ 1639.068307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1639.113880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1639.126864] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 1640.009407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1640.025290] LNet: Added LNI 192.168.202.41@tcp1 [8/256/0/180] [ 1641.244634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.41@tcp [ 1641.316947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1641.480248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1641.534468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1641.647687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1641.678666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1641.750879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1641.825954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.41@tcp --health 0 [ 1642.208610] LNet: There was an unexpected network error while writing to 192.168.202.41: rc = -22 [ 1642.214555] LNet: Skipped 3 previous similar messages [ 1643.125727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1653.636099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1653.680419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1653.736100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1662.634929] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 20:58:18 (1781225898) [ 1663.022729] LNet: 68671:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1663.029271] LNetError: 68671:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1663.034796] LNetError: 68671:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1664.096151] LNet: Removed LNI 192.168.202.41@tcp [ 1664.121244] LNet: Removed LNI 192.168.202.41@tcp1 [ 1664.877914] Key type .llcrypt unregistered [ 1664.882499] Key type ._llcrypt unregistered [ 1673.014781] Key type ._llcrypt registered [ 1673.019984] Key type .llcrypt registered [ 1673.446884] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1673.483034] alg: No test for adler32 (adler32-zlib) [ 1674.461422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1674.503271] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 1685.261566] Lustre: DEBUG MARKER: Initial discovery [ 1685.375670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.141@tcp [ 1688.266710] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1688.529394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1688.634747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.141@tcp [ 1688.670722] LNet: There was an unexpected network error while writing to 192.168.202.141: rc = -22 [ 1688.694554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1688.747957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.202.141@tcp -r 1 -m GET -l 3 [ 1688.805647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.202.141@tcp -r 1 -m GET -e local_error [ 1688.865915] LNet: 1 local NIs in recovery (showing 1): 192.168.202.41@tcp [ 1688.876330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.202.141@tcp -r 1 -m PUT -l 6 [ 1690.561159] Lustre: DEBUG MARKER: Force oleg241-client.virtnet to discover 192.168.202.141@tcp (in background) [ 1690.601132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.141@tcp [ 1692.082181] Lustre: DEBUG MARKER: Force oleg241-server to discover 192.168.202.41@tcp [ 1693.664850] LNet: There was an unexpected network error while writing to 192.168.202.141: rc = -22 [ 1694.427787] Lustre: DEBUG MARKER: Wait for 69426 [ 1696.236756] Lustre: DEBUG MARKER: Finished wait on 69426 [ 1696.298697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1696.346499] LNet: There was an unexpected network error while writing to 192.168.202.141: rc = -22 [ 1696.349201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1696.728560] LNet: 69883:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1696.739833] LNetError: 69883:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1697.771757] LNet: Removed LNI 192.168.202.41@tcp [ 1698.375458] Key type .llcrypt unregistered [ 1698.376923] Key type ._llcrypt unregistered [ 1719.113344] Key type ._llcrypt registered [ 1719.115221] Key type .llcrypt registered [ 1720.525706] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 20:59:16 (1781225956) [ 1721.301549] Key type .llcrypt unregistered [ 1721.304171] Key type ._llcrypt unregistered [ 1728.923308] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1729.107862] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1729.109794] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1729.111378] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1730.760844] Key type ._llcrypt registered [ 1730.763970] Key type .llcrypt registered [ 1731.007060] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1731.021810] alg: No test for adler32 (adler32-zlib) [ 1732.014992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1732.081923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1732.130211] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 1733.159897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1733.190314] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1745.547208] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 20:59:41 (1781225981) [ 1745.763552] LNet: 71951:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1745.773430] LNetError: 71951:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1745.775749] LNet: Removed LNI 192.168.202.41@tcp [ 1745.799421] LNet: Removed LNI 10.1.2.3@tcp [ 1746.480459] Key type .llcrypt unregistered [ 1746.482429] Key type ._llcrypt unregistered [ 1754.852266] Key type ._llcrypt registered [ 1754.854734] Key type .llcrypt registered [ 1754.975309] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1754.990309] alg: No test for adler32 (adler32-zlib) [ 1755.972421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1756.013217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1756.025959] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 1756.499392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1756.510820] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1757.165380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.202.41@tcp [ 1758.240280] LNetError: Unexpected error -22 connecting to 192.168.202.41@tcp at host 192.168.202.41:988 [ 1765.668904] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 21:00:01 (1781226001) [ 1765.920464] LNet: 73088:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1765.924688] LNetError: 73088:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1765.928925] LNet: Removed LNI 192.168.202.41@tcp [ 1765.944223] LNet: Removed LNI 10.1.2.3@tcp [ 1766.645446] Key type .llcrypt unregistered [ 1766.652329] Key type ._llcrypt unregistered [ 1772.272146] Key type ._llcrypt registered [ 1772.273265] Key type .llcrypt registered [ 1772.609599] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1772.642461] alg: No test for adler32 (adler32-zlib) [ 1773.537262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1773.571792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1773.587703] LNet: Added LNI 192.168.202.41@tcp1 [8/256/0/180] [ 1774.160336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1774.185046] LNet: Added LNI 192.168.202.41@tcp2 [8/256/0/180] [ 1774.814181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.202.41@tcp1 --nid 192.168.202.41@tcp2 [ 1781.088367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.41@tcp1 192.168.202.41@tcp2 [ 1781.120324] LNetError: 73923:0:(lib-move.c:2007:lnet_handle_find_routed_path()) Unable to send message from 192.168.202.41@tcp1 to 192.168.202.41@tcp2 - Route table may be misconfigured [ 1781.175866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.41@tcp2 192.168.202.41@tcp1 [ 1789.710526] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 21:00:25 (1781226025) [ 1789.880586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1789.912631] LNet: 74492:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1789.933066] LNetError: 74492:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1789.942322] LNet: Removed LNI 192.168.202.41@tcp1 [ 1791.016932] LNet: Removed LNI 192.168.202.41@tcp2 [ 1791.063740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1791.225269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1791.258507] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 1792.305377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1792.330793] LNet: Added LNI 192.168.202.41@tcp1 [8/256/0/180] [ 1793.380494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.41@tcp [ 1793.401792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.41@tcp 192.168.202.41@tcp [ 1793.438295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.41@tcp1 192.168.202.41@tcp1 [ 1793.487537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.41@tcp -d 192.168.202.41@tcp -e network_timeout [ 1793.519848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.41@tcp -d 192.168.202.41@tcp1 -e network_timeout [ 1793.550528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.41@tcp1 -d 192.168.202.41@tcp -e network_timeout [ 1793.584229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.41@tcp1 -d 192.168.202.41@tcp1 -e network_timeout [ 1793.644303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.41@tcp [ 1793.664624] LNet: There was an unexpected network error while writing to 192.168.202.41: rc = -22 [ 1793.702353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1793.870330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1794.062186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1794.212068] LNet: 1 local NIs in recovery (showing 1): 192.168.202.41@tcp [ 1804.312939] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 21:00:39 (1781226039) [ 1804.436150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1804.480627] LNet: 75360:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1804.486040] LNetError: 75360:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1804.490375] LNetError: 75360:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1804.497238] LNet: Removed LNI 192.168.202.41@tcp [ 1805.544977] LNet: Removed LNI 192.168.202.41@tcp1 [ 1805.599781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1805.717269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1806.038523] LNet: 75397:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1806.383321] Key type .llcrypt unregistered [ 1806.386587] Key type ._llcrypt unregistered [ 1821.847792] Key type ._llcrypt registered [ 1821.849112] Key type .llcrypt registered [ 1823.166677] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 21:00:59 (1781226059) [ 1823.463431] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1823.494220] alg: No test for adler32 (adler32-zlib) [ 1824.601331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1826.551887] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1827.892185] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 21:01:04 (1781226064) [ 1827.979123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1828.005142] LNet: 76501:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1828.025755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1828.076988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1828.087982] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 1829.055522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1829.076492] LNet: Added LNI 192.168.202.41@tcp1 [8/256/0/180] [ 1829.867750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.41@tcp [ 1829.912952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.41@tcp1 [ 1829.952483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.41@tcp1 [ 1836.142502] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 21:01:12 (1781226072) [ 1838.333173] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg241-server [ 1840.515757] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 21:01:16 (1781226076) [ 1842.949734] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg241-server [ 1845.128623] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 21:01:20 (1781226080) [ 1847.199461] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg241-server [ 1849.382102] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 21:01:25 (1781226085) [ 1851.884111] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg241-server [ 1853.622768] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 21:01:29 (1781226089) [ 1855.879615] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg241-server [ 1857.975933] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 21:01:33 (1781226093) [ 1860.195795] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg241-server [ 1862.128533] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 21:01:38 (1781226098) [ 1864.660793] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg241-server [ 1866.873866] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 21:01:42 (1781226102) [ 1868.756660] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg241-server [ 1871.092675] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 21:01:46 (1781226106) [ 1873.749672] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg241-server [ 1875.886138] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 21:01:51 (1781226111) [ 1876.013662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1876.062548] LNet: 80380:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1876.068821] LNetError: 80380:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1876.089164] LNet: Removed LNI 192.168.202.41@tcp [ 1877.158848] LNet: Removed LNI 192.168.202.41@tcp1 [ 1877.227193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1877.360760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1877.416806] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 1878.273596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1878.352476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.41@tcp [ 1878.586684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1878.610293] LNet: 80534:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1878.621777] LNetError: 80534:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1878.633763] LNetError: 80534:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1878.652187] LNet: Removed LNI 192.168.202.41@tcp [ 1878.695517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1878.756447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1878.788646] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 1879.649408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.41@tcp [ 1879.818568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1879.841290] LNetError: 80679:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1880.873619] LNet: Removed LNI 192.168.202.41@tcp [ 1880.957517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1881.026917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1881.046252] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 1882.073504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1882.165625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.41@tcp [ 1882.395958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1882.445318] LNet: 80827:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1882.453290] LNet: 80827:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1882.459953] LNetError: 80827:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1882.495060] LNet: Removed LNI 192.168.202.41@tcp [ 1882.554662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1882.595421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1882.616323] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 1883.483417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.41@tcp [ 1883.697715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1884.779175] LNet: Removed LNI 192.168.202.41@tcp [ 1884.839740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1884.929800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1884.957769] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 1885.740640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1885.898949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.41@tcp [ 1886.221040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1887.270746] LNet: Removed LNI 192.168.202.41@tcp [ 1887.289401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1887.387977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1887.438440] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 1888.390539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.41@tcp [ 1888.551620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1888.586608] LNet: 81265:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1888.589242] LNet: 81265:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1888.598580] LNetError: 81265:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1888.602748] LNetError: 81265:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1889.639327] LNet: Removed LNI 192.168.202.41@tcp [ 1889.679162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1889.765614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1889.810604] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 1890.815739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1890.915756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.41@tcp [ 1892.320272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1893.421487] LNet: Removed LNI 192.168.202.41@tcp [ 1893.484682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1893.552786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1893.591406] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 1894.526545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.41@tcp [ 1894.707674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1895.784551] LNet: Removed LNI 192.168.202.41@tcp [ 1895.833574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1895.922296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1895.942383] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 1896.856691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1896.954325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.41@tcp [ 1897.104059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1897.133107] LNet: 81714:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1897.144112] LNet: 81714:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1897.148857] LNetError: 81714:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1897.154766] LNetError: 81714:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1897.167880] LNet: Removed LNI 192.168.202.41@tcp [ 1897.195455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1897.267733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1897.326162] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 1898.326643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.41@tcp [ 1898.525566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1899.560293] LNet: Removed LNI 192.168.202.41@tcp [ 1899.603915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1899.676672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1899.701819] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 1900.486898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1900.556157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.41@tcp [ 1900.804521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1901.863231] LNet: Removed LNI 192.168.202.41@tcp [ 1901.895879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1901.968044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1902.009151] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 1902.694666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.41@tcp [ 1902.996568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1904.106936] LNet: Removed LNI 192.168.202.41@tcp [ 1904.157659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1904.229676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1904.246676] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 1904.985795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1905.046995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1905.084393] LNet: Removed LNI 192.168.202.41@tcp [ 1905.129428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1905.278589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1905.324857] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 1906.123856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1916.098459] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 21:02:31 (1781226151) [ 1916.217440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1916.304957] LNet: 82983:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1916.311807] LNet: 82983:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1916.317939] LNetError: 82983:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1916.323615] LNetError: 82983:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1916.368893] LNet: Removed LNI 192.168.202.41@tcp [ 1916.407781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1916.462837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1916.510026] LNet: Added LNI 192.168.202.41@tcp231 [8/256/0/180] [ 1919.320310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1919.383914] LNet: Removed LNI 192.168.202.41@tcp231 [ 1919.406830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1919.617089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1919.711277] LNet: Added LNI 192.168.202.41@tcp231 [8/256/0/180] [ 1919.729645] LNet: 83135:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1920.903979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1920.924873] LNet: Removed LNI 192.168.202.41@tcp231 [ 1920.935910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1920.967361] LNet: Added LNI 192.168.202.41@tcp231 [8/256/0/180] [ 1922.843634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1922.895038] LNet: Removed LNI 192.168.202.41@tcp231 [ 1922.938315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1923.033901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1923.061469] LNet: Added LNI 192.168.202.41@tcp231 [8/256/0/0] [ 1923.066754] LNet: 83288:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1923.571980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1923.618814] LNet: Removed LNI 192.168.202.41@tcp231 [ 1923.635918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1923.674374] LNet: Added LNI 192.168.202.41@tcp231 [8/256/0/0] [ 1925.891242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1925.923129] LNet: Removed LNI 192.168.202.41@tcp231 [ 1925.940991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1926.030470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1926.055380] LNet: Added LNI 192.168.202.41@tcp231 [8/256/0/60] [ 1926.060124] LNet: 83439:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1926.860710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1926.914643] LNet: Removed LNI 192.168.202.41@tcp231 [ 1926.941229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1927.046532] LNet: Added LNI 192.168.202.41@tcp231 [8/256/0/60] [ 1928.502715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1928.522839] LNet: Removed LNI 192.168.202.41@tcp231 [ 1928.537035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1928.593709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1928.614101] LNet: Added LNI 192.168.202.41@tcp231 [8/256/0/180] [ 1928.621130] LNet: 83591:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1928.804325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1928.825612] LNet: Removed LNI 192.168.202.41@tcp231 [ 1928.836056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1928.849290] LNet: Added LNI 192.168.202.41@tcp231 [8/256/0/180] [ 1937.830194] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 21:02:53 (1781226173) [ 1937.934078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1938.012344] LNet: Removed LNI 192.168.202.41@tcp231 [ 1938.072496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1938.190834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1938.271061] LNet: Added LNI 192.168.202.41@tcp232 [8/256/0/180] [ 1939.054438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1939.085743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1939.121239] LNet: Removed LNI 192.168.202.41@tcp232 [ 1939.143350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1939.166468] LNet: Added LNI 192.168.202.41@tcp232 [8/256/0/180] [ 1939.752191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1939.771628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1939.791991] LNet: Removed LNI 192.168.202.41@tcp232 [ 1939.802255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1939.836459] LNet: Added LNI 192.168.202.41@tcp232 [8/256/0/180] [ 1940.295817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1940.317827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1940.340484] LNet: Removed LNI 192.168.202.41@tcp232 [ 1940.355020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1940.369818] LNet: Added LNI 192.168.202.41@tcp232 [8/256/0/180] [ 1941.190210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1941.223916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1941.258813] LNet: Removed LNI 192.168.202.41@tcp232 [ 1941.285423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1951.378822] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 21:03:05 (1781226185) [ 1952.505437] LNet: 85431:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1952.514999] LNet: 85431:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 1953.875988] Key type .llcrypt unregistered [ 1953.889246] Key type ._llcrypt unregistered [ 1962.878632] Key type ._llcrypt registered [ 1962.880165] Key type .llcrypt registered [ 1963.047611] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1963.063278] alg: No test for adler32 (adler32-zlib) [ 1963.919818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1963.947821] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 1974.082870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.141@tcp [ 1986.680302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1986.711126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1986.731747] LNet: Added LNI 192.168.202.41@tcp1 [8/256/0/180] [ 1988.286328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.41@tcp -d 192.168.202.141@tcp -m GET -e local_interrupt -r 2 [ 1988.343790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.41@tcp -d 192.168.202.141@tcp1 -m GET -e local_interrupt -r 2 [ 1988.396133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.41@tcp -d 192.168.202.41@tcp -m GET -e local_interrupt -r 2 [ 1988.454655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.41@tcp -d 192.168.202.41@tcp1 -m GET -e local_interrupt -r 2 [ 1988.507412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.41@tcp1 -d 192.168.202.141@tcp -m GET -e local_interrupt -r 2 [ 1988.537995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.41@tcp1 -d 192.168.202.141@tcp1 -m GET -e local_interrupt -r 2 [ 1988.618656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.41@tcp1 -d 192.168.202.41@tcp -m GET -e local_interrupt -r 2 [ 1988.656488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.41@tcp1 -d 192.168.202.41@tcp1 -m GET -e local_interrupt -r 2 [ 1988.699861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.41@tcp 192.168.202.141@tcp [ 1988.725405] LNet: There was an unexpected network error while writing to 192.168.202.141: rc = -22 [ 1988.734184] LNet: 1 local NIs in recovery (showing 1): 192.168.202.41@tcp [ 1988.913186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1988.934430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1988.961294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1988.978347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1988.996307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1989.027684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1989.059884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1989.087274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1989.102832] LNet: 86094:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1989.125927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1989.156173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1989.193507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1989.225822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1989.241514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1989.264628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1989.767735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.41@tcp -d 192.168.202.141@tcp -m GET -e local_dropped -r 2 [ 1989.824848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.41@tcp -d 192.168.202.141@tcp1 -m GET -e local_dropped -r 2 [ 1989.887642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.41@tcp -d 192.168.202.41@tcp -m GET -e local_dropped -r 2 [ 1989.939266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.41@tcp -d 192.168.202.41@tcp1 -m GET -e local_dropped -r 2 [ 1989.992835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.41@tcp1 -d 192.168.202.141@tcp -m GET -e local_dropped -r 2 [ 1990.054569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.41@tcp1 -d 192.168.202.141@tcp1 -m GET -e local_dropped -r 2 [ 1990.112692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.41@tcp1 -d 192.168.202.41@tcp -m GET -e local_dropped -r 2 [ 1990.175975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.41@tcp1 -d 192.168.202.41@tcp1 -m GET -e local_dropped -r 2 [ 1990.242749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.41@tcp 192.168.202.141@tcp [ 1990.289911] LNet: There was an unexpected network error while writing to 192.168.202.141: rc = -22 [ 1990.712468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1990.765353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1990.832329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1990.900330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1990.947870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1991.018643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1991.085988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1991.176075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1991.220646] LNet: 86215:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1991.282696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1991.311515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1991.338929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1991.386606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1991.442705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1991.497137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1992.378595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.41@tcp -d 192.168.202.141@tcp -m GET -e local_aborted -r 2 [ 1992.419510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.41@tcp -d 192.168.202.141@tcp1 -m GET -e local_aborted -r 2 [ 1992.484665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.41@tcp -d 192.168.202.41@tcp -m GET -e local_aborted -r 2 [ 1992.588300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.41@tcp -d 192.168.202.41@tcp1 -m GET -e local_aborted -r 2 [ 1992.673307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.41@tcp1 -d 192.168.202.141@tcp -m GET -e local_aborted -r 2 [ 1992.804661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.41@tcp1 -d 192.168.202.141@tcp1 -m GET -e local_aborted -r 2 [ 1992.890437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.41@tcp1 -d 192.168.202.41@tcp -m GET -e local_aborted -r 2 [ 1993.018406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.41@tcp1 -d 192.168.202.41@tcp1 -m GET -e local_aborted -r 2 [ 1993.100721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.41@tcp 192.168.202.141@tcp [ 1993.119595] LNet: There was an unexpected network error while writing to 192.168.202.141: rc = -22 [ 1993.300669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1993.332932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1993.364528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1993.407696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1993.447343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1993.492409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1993.530478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1993.579085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1993.610726] LNet: 86335:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1993.625957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1993.661535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1993.731294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1993.802275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1993.844288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1993.890035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1994.319462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.41@tcp -d 192.168.202.141@tcp -m GET -e local_no_route -r 2 [ 1994.356056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.41@tcp -d 192.168.202.141@tcp1 -m GET -e local_no_route -r 2 [ 1994.385058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.41@tcp -d 192.168.202.41@tcp -m GET -e local_no_route -r 2 [ 1994.418691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.41@tcp -d 192.168.202.41@tcp1 -m GET -e local_no_route -r 2 [ 1994.448567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.41@tcp1 -d 192.168.202.141@tcp -m GET -e local_no_route -r 2 [ 1994.489547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.41@tcp1 -d 192.168.202.141@tcp1 -m GET -e local_no_route -r 2 [ 1994.530473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.41@tcp1 -d 192.168.202.41@tcp -m GET -e local_no_route -r 2 [ 1994.557434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.41@tcp1 -d 192.168.202.41@tcp1 -m GET -e local_no_route -r 2 [ 1994.575744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.41@tcp 192.168.202.141@tcp [ 1994.744263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1994.763324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1994.779923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1994.796537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1994.813701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1994.832189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1994.864916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1994.890771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1994.909838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1994.934833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1994.951837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1994.971086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1995.013336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1995.061576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1995.651866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.41@tcp -d 192.168.202.141@tcp -m GET -e local_timeout -r 2 [ 1995.685910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.41@tcp -d 192.168.202.141@tcp1 -m GET -e local_timeout -r 2 [ 1995.727618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.41@tcp -d 192.168.202.41@tcp -m GET -e local_timeout -r 2 [ 1995.795628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.41@tcp -d 192.168.202.41@tcp1 -m GET -e local_timeout -r 2 [ 1995.845974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.41@tcp1 -d 192.168.202.141@tcp -m GET -e local_timeout -r 2 [ 1995.879948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.41@tcp1 -d 192.168.202.141@tcp1 -m GET -e local_timeout -r 2 [ 1995.913336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.41@tcp1 -d 192.168.202.41@tcp -m GET -e local_timeout -r 2 [ 1995.958348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.41@tcp1 -d 192.168.202.41@tcp1 -m GET -e local_timeout -r 2 [ 1996.019433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.41@tcp 192.168.202.141@tcp [ 1996.091031] LNet: There was an unexpected network error while writing to 192.168.202.141: rc = -22 [ 1996.102355] LNet: Skipped 1 previous similar message [ 1996.446480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1996.484381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1996.539274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1996.604114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1996.648892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1996.667294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1996.686295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1996.712161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1996.731477] LNet: 86576:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1996.734429] LNet: 86576:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1996.764115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1996.793640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1996.820221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1996.861562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1996.896313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1996.926620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1997.451862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.41@tcp -d 192.168.202.141@tcp -m GET -e remote_dropped -r 2 [ 1997.477413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.41@tcp -d 192.168.202.141@tcp1 -m GET -e remote_dropped -r 2 [ 1997.509189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.41@tcp -d 192.168.202.41@tcp -m GET -e remote_dropped -r 2 [ 1997.537252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.41@tcp -d 192.168.202.41@tcp1 -m GET -e remote_dropped -r 2 [ 1997.582097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.41@tcp1 -d 192.168.202.141@tcp -m GET -e remote_dropped -r 2 [ 1997.615400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.41@tcp1 -d 192.168.202.141@tcp1 -m GET -e remote_dropped -r 2 [ 1997.645535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.41@tcp1 -d 192.168.202.41@tcp -m GET -e remote_dropped -r 2 [ 1997.668466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.41@tcp1 -d 192.168.202.41@tcp1 -m GET -e remote_dropped -r 2 [ 1997.687921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.41@tcp 192.168.202.141@tcp [ 1997.705785] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.141@tcp [ 1997.930590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1997.967429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1997.995434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1998.030277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1998.068615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1998.113581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1998.145773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1998.174766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1998.221505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1998.267527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1998.304460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1998.334290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1998.376711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1998.410652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2004.272314] LNet: 86770:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2004.278546] LNetError: 86770:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2004.291726] LNet: Removed LNI 192.168.202.41@tcp [ 2004.310556] LNet: Removed LNI 192.168.202.41@tcp1 [ 2004.754340] Key type .llcrypt unregistered [ 2004.756317] Key type ._llcrypt unregistered [ 2019.414552] Key type ._llcrypt registered [ 2019.424504] Key type .llcrypt registered [ 2020.998371] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 21:04:16 (1781226256) [ 2021.528487] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2021.546979] alg: No test for adler32 (adler32-zlib) [ 2022.559218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2022.630377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2022.651259] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 2023.644957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 2023.720902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.41@tcp [ 2023.772802] LNetError: 87748:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.202.41@tcp to recovery [ 2023.894227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.202.41@tcp [ 2024.022816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2030.477769] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 21:04:26 (1781226266) [ 2030.638466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2030.686375] LNet: 88337:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2030.702330] LNetError: 88337:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2030.724226] LNet: Removed LNI 192.168.202.41@tcp [ 2030.743397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2030.797292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2030.818813] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 2031.618663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2031.631101] LNet: Added LNI 192.168.202.41@tcp1 [8/256/0/180] [ 2032.979644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.202.41@tcp [ 2033.013417] LNetError: 88603:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.202.41@tcp to recovery [ 2033.888415] LNet: 1 local NIs in recovery (showing 1): 192.168.202.41@tcp [ 2033.889930] LNet: There was an unexpected network error while writing to 192.168.202.41: rc = -22 [ 2034.428423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.202.41@tcp [ 2045.466174] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 21:04:41 (1781226281) [ 2048.252895] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg241-server [ 2051.682326] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 21:04:46 (1781226286) [ 2054.623124] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg241-server [ 2056.621359] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 21:04:52 (1781226292) [ 2058.859977] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg241-server [ 2061.248444] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 21:04:56 (1781226296) [ 2061.353428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2061.382143] LNet: 90228:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2061.402225] LNetError: 90228:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2061.427675] LNet: Removed LNI 192.168.202.41@tcp [ 2061.457412] LNet: Removed LNI 192.168.202.41@tcp1 [ 2061.539928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2061.723406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2061.796761] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 2063.103651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 2063.190714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 2063.251957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2063.362647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 2063.449951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2063.547158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 2071.632950] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 21:05:07 (1781226307) [ 2071.851676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2071.905652] LNet: 90959:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2071.915275] LNetError: 90959:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2071.931579] LNetError: 90959:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2071.951823] LNet: Removed LNI 192.168.202.41@tcp [ 2071.974904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2072.036912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2072.061083] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 2079.677730] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 21:05:15 (1781226315) [ 2079.986640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2080.000634] LNet: 91665:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2080.005706] LNetError: 91665:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2080.015503] LNet: Removed LNI 192.168.202.41@tcp [ 2080.042791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2080.114418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2080.141105] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 2087.597229] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 21:05:23 (1781226323) [ 2089.043323] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 2090.669972] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 21:05:26 (1781226326) [ 2091.290971] LNet: 92700:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2091.295820] LNetError: 92700:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2091.308313] LNet: Removed LNI 192.168.202.41@tcp [ 2091.918742] Key type .llcrypt unregistered [ 2091.921791] Key type ._llcrypt unregistered [ 2099.157330] Key type ._llcrypt registered [ 2099.158586] Key type .llcrypt registered [ 2099.394327] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2099.412909] alg: No test for adler32 (adler32-zlib) [ 2100.423335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2100.459733] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 2109.224656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.141@tcp [ 2114.633784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.202.141@tcp [ 2115.146418] LNet: 93119:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2115.154592] LNetError: 93119:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2115.172676] LNet: Removed LNI 192.168.202.41@tcp [ 2115.764452] Key type .llcrypt unregistered [ 2115.767314] Key type ._llcrypt unregistered [ 2128.200867] Key type ._llcrypt registered [ 2128.203384] Key type .llcrypt registered [ 2129.575450] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 2131.233165] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 2132.866739] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 21:06:09 (1781226369) [ 2134.534702] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 2136.488079] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 21:06:12 (1781226372) [ 2138.478556] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg241-server [ 2140.389716] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 21:06:16 (1781226376) [ 2142.316239] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg241-server [ 2144.789941] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 21:06:20 (1781226380) [ 2145.921764] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2145.934074] alg: No test for adler32 (adler32-zlib) [ 2147.245557] Lustre: Lustre: Build Version: 2.17.53_70_g68e5078 [ 2147.568247] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 2149.392191] Key type lgssc registered [ 2151.959651] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2181.292037] Key type lgssc unregistered [ 2181.489600] LNet: 96297:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2181.495919] LNetError: 96297:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2181.507563] LNet: Removed LNI 192.168.202.41@tcp [ 2182.102850] Key type .llcrypt unregistered [ 2182.105768] Key type ._llcrypt unregistered [ 2199.954413] Key type ._llcrypt registered [ 2199.958593] Key type .llcrypt registered [ 2201.418975] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 21:07:17 (1781226437) [ 2201.711583] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2201.727315] alg: No test for adler32 (adler32-zlib) [ 2202.612769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2202.678492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2202.714318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2202.764092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2202.806637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2209.169385] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 21:07:25 (1781226445) [ 2209.412170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2209.450368] LNet: 97712:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2209.483701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2209.546381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2209.572253] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 2210.580183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2210.598512] LNet: 97856:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2210.604687] LNetError: 97856:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2210.618960] LNet: Removed LNI 192.168.202.41@tcp [ 2210.636563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2210.707422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2210.744667] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 2211.239461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2211.266377] LNetError: 97884:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2211.283146] LNet: Removed LNI 192.168.202.41@tcp [ 2211.312277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2211.376187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2211.403389] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 2218.657504] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 21:07:35 (1781226455) [ 2218.866276] LNet: 98473:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2218.869616] LNet: 98473:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2218.873856] LNetError: 98473:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2218.888753] LNet: Removed LNI 192.168.202.41@tcp [ 2219.563575] Key type .llcrypt unregistered [ 2219.567615] Key type ._llcrypt unregistered [ 2227.188343] Key type ._llcrypt registered [ 2227.192254] Key type .llcrypt registered [ 2227.408559] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2227.439302] alg: No test for adler32 (adler32-zlib) [ 2228.273809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2228.338752] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 2228.344802] LNet: *** cfs_fail_loc=e003, val=100*** [ 2229.801624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.41@tcp [ 2229.840357] LNet: *** cfs_fail_loc=e003, val=100*** [ 2229.844218] LNet: Skipped 2 previous similar messages [ 2229.857899] Ping reply received with mapping: [ 2229.859569] found metadata reply NI[192.168.202.41@tcp] [ 2229.866396] 29 ca a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2229.871294] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2229.884989] 5a 5a 5a 5a 5a 5a 5a 5a [ 2230.219803] LNet: 98872:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2230.237923] LNetError: 98872:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2230.270945] LNet: Removed LNI 192.168.202.41@tcp [ 2231.259337] Key type .llcrypt unregistered [ 2231.260409] Key type ._llcrypt unregistered [ 2248.000309] Key type ._llcrypt registered [ 2248.003984] Key type .llcrypt registered [ 2250.510268] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 21:08:05 (1781226485) [ 2251.638963] Key type .llcrypt unregistered [ 2251.654102] Key type ._llcrypt unregistered [ 2261.535111] Key type ._llcrypt registered [ 2261.537267] Key type .llcrypt registered [ 2261.916947] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2261.961145] alg: No test for adler32 (adler32-zlib) [ 2263.204467] LNetError: 99921:0:(api-ni.c:2729:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2263.779087] Key type .llcrypt unregistered [ 2263.780851] Key type ._llcrypt unregistered [ 2270.912643] Key type ._llcrypt registered [ 2270.920451] Key type .llcrypt registered [ 2272.303490] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 21:08:28 (1781226508) [ 2272.714345] Key type .llcrypt unregistered [ 2272.716069] Key type ._llcrypt unregistered [ 2279.961684] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2279.973742] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2282.683536] Key type ._llcrypt registered [ 2282.686247] Key type .llcrypt registered [ 2282.868520] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2282.894548] alg: No test for adler32 (adler32-zlib) [ 2283.957172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2284.075946] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2295.324738] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 21:08:50 (1781226530) [ 2295.426201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2295.485255] LNet: 101695:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2295.492870] LNetError: 101695:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2295.531561] LNet: Removed LNI 10.1.2.3@tcp [ 2295.581498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2296.317443] Key type .llcrypt unregistered [ 2296.321307] Key type ._llcrypt unregistered [ 2304.059975] Key type ._llcrypt registered [ 2304.063562] Key type .llcrypt registered [ 2304.349602] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2304.381355] alg: No test for adler32 (adler32-zlib) [ 2305.394042] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 2305.503594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.41@tcp [ 2305.785045] LNet: 101991:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2305.802665] LNetError: 101991:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2305.816030] LNet: Removed LNI 192.168.202.41@tcp [ 2306.700847] Key type .llcrypt unregistered [ 2306.709134] Key type ._llcrypt unregistered [ 2319.349696] Key type ._llcrypt registered [ 2319.351032] Key type .llcrypt registered [ 2321.346550] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 21:09:16 (1781226556) [ 2322.534334] Key type .llcrypt unregistered [ 2322.541094] Key type ._llcrypt unregistered [ 2331.214461] Key type ._llcrypt registered [ 2331.221780] Key type .llcrypt registered [ 2331.403463] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2331.422134] alg: No test for adler32 (adler32-zlib) [ 2340.204074] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 21:09:36 (1781226576) [ 2340.399673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2340.450414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2340.541484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2340.583244] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 2341.493379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2341.562335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2341.610511] LNet: Removed LNI 192.168.202.41@tcp [ 2341.964574] LNet: 103735:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2342.439418] Key type .llcrypt unregistered [ 2342.445422] Key type ._llcrypt unregistered [ 2356.162974] Key type ._llcrypt registered [ 2356.169169] Key type .llcrypt registered [ 2357.302688] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 21:09:53 (1781226593) [ 2357.882390] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2357.891879] alg: No test for adler32 (adler32-zlib) [ 2358.828877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2358.891781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2358.905808] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 2359.692245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.41@tcp [ 2359.734423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.202.41@tcp [ 2359.756961] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2359.763859] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2359.769260] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2359, 0, 0 [ 2359.773923] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2359.776898] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2359.784493] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2359.787340] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2359, 0, 0 [ 2359.790154] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2359.795523] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2359.798913] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2359.801640] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2359, 0, 2409 [ 2359.806925] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2359.809850] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2359.816790] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2359.822264] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2359, 0, 2409 [ 2359.829759] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2366.971337] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 21:10:03 (1781226603) [ 2367.638870] LNet: 105275:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2367.650221] LNetError: 105275:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2367.689444] LNet: Removed LNI 192.168.202.41@tcp [ 2368.761457] Key type .llcrypt unregistered [ 2368.775567] Key type ._llcrypt unregistered [ 2376.005148] Key type ._llcrypt registered [ 2376.006754] Key type .llcrypt registered [ 2376.221622] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2376.251377] alg: No test for adler32 (adler32-zlib) [ 2377.191967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2377.232964] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 2385.155738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.141@tcp [ 2394.890619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2394.953985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2394.983099] LNet: Added LNI 192.168.202.41@tcp1 [8/256/0/180] [ 2396.402448] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2396.616775] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2396.638159] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2396.640524] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2399.235960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2399.273730] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2415.656359] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 21:10:51 (1781226651) [ 2415.835277] LNet: 106779:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2415.849444] LNetError: 106779:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2415.856525] LNet: Removed LNI 192.168.202.41@tcp [ 2416.928235] LNet: Removed LNI 192.168.202.41@tcp1 [ 2416.930199] LNetError: 106779:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2416.954899] LNetError: 106779:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2416.985965] LNet: Removed LNI 10.1.2.3@tcp99 [ 2418.156260] Key type .llcrypt unregistered [ 2418.160097] Key type ._llcrypt unregistered [ 2425.539980] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2425.555086] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2427.166931] Key type ._llcrypt registered [ 2427.168392] Key type .llcrypt registered [ 2427.373505] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2427.396438] alg: No test for adler32 (adler32-zlib) [ 2428.317296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2428.369692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2428.434661] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 2429.488770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2429.540457] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2433.015410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2433.081295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2433.136500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2433.167444] LNetError: 107523:0:(peer.c:2175:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2433.198611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2433.237623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.41@tcp [ 2433.684415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2433.784682] LNet: 107543:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2433.796177] LNetError: 107543:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2434.848243] LNet: Removed LNI 192.168.202.41@tcp [ 2434.853112] LNetError: 107543:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2434.867184] LNet: Removed LNI 10.1.2.3@tcp [ 2435.430477] Key type .llcrypt unregistered [ 2435.432805] Key type ._llcrypt unregistered [ 2448.190087] Key type ._llcrypt registered [ 2448.192956] Key type .llcrypt registered [ 2449.631766] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 21:11:25 (1781226685) [ 2450.062550] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2450.072631] alg: No test for adler32 (adler32-zlib) [ 2450.948843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2451.031211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2451.072195] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 2451.706938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.41@tcp [ 2451.754671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg241-client@tcp [ 2459.235041] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 21:11:34 (1781226694) [ 2459.347276] LNet: 109070:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2459.355666] LNetError: 109070:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2459.379651] LNet: Removed LNI 192.168.202.41@tcp [ 2460.397605] Key type .llcrypt unregistered [ 2460.399803] Key type ._llcrypt unregistered [ 2471.441415] Key type ._llcrypt registered [ 2471.451604] Key type .llcrypt registered [ 2471.919257] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2471.951460] alg: No test for adler32 (adler32-zlib) [ 2472.891940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2472.920160] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 2472.998817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2479.692788] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 21:11:55 (1781226715) [ 2481.364478] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2483.430331] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 21:11:59 (1781226719) [ 2484.935355] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2487.296271] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 21:12:02 (1781226722) [ 2487.608456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2487.647927] LNet: 110499:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2487.660085] LNetError: 110499:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2487.680552] LNet: Removed LNI 192.168.202.41@tcp [ 2487.727919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2487.782740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2487.802774] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 2488.603911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2488.667136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2496.998857] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 21:12:12 (1781226732) [ 2497.314382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2497.355466] LNet: 111208:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2497.372626] LNetError: 111208:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2497.419088] LNet: Removed LNI 192.168.202.41@tcp [ 2497.467111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2497.541987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2497.568821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2498.154590] Key type .llcrypt unregistered [ 2498.157818] Key type ._llcrypt unregistered [ 2511.892952] Key type ._llcrypt registered [ 2511.895176] Key type .llcrypt registered [ 2513.518497] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 21:12:29 (1781226749) [ 2513.931540] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2513.954636] alg: No test for adler32 (adler32-zlib) [ 2515.067497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2515.188095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2515.237448] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 2516.189080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2516.260446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.41@tcp [ 2524.218484] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 21:12:39 (1781226759) [ 2524.381718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2524.434120] LNet: 112759:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2524.449388] LNetError: 112759:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2524.475523] LNet: Removed LNI 192.168.202.41@tcp [ 2524.510597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2524.658234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2524.705529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2531.987365] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 21:12:48 (1781226768) [ 2532.442388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2532.488237] LNet: 113345:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2532.543601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2532.681202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2533.282183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2533.321945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2533.446116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2534.283438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2534.368599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2534.450696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2534.493942] LNet: 113378:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2534.519805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2534.662589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2534.969856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2535.001330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2535.085907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2535.805957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2535.839369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2544.440576] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 21:13:00 (1781226780) [ 2544.527188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2544.563868] LNet: 113971:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2544.582438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2544.627713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2544.646761] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 2545.485268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2545.547558] LNet: Added LNI 192.168.202.41@tcp2 [8/256/0/180] [ 2546.409588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2546.485520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.41@tcp [ 2546.744940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2546.761843] LNetError: 114258:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2546.766168] LNet: Removed LNI 192.168.202.41@tcp [ 2546.773910] LNet: Removed LNI 192.168.202.41@tcp2 [ 2546.786845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2546.836934] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 2546.841856] LNet: Added LNI 192.168.202.41@tcp2 [8/256/0/180] [ 2546.847482] LNet: 114264:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2547.307307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.41@tcp [ 2547.546495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2548.576174] LNet: Removed LNI 192.168.202.41@tcp [ 2548.578366] LNetError: 114301:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2548.584237] LNetError: 114301:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2548.597137] LNet: Removed LNI 192.168.202.41@tcp2 [ 2548.630392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2548.707493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2548.724083] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 2549.186487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2549.208043] LNet: Added LNI 192.168.202.41@tcp2 [8/256/0/180] [ 2549.900324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.41@tcp [ 2549.986523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2550.055525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.41@tcp [ 2550.248947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2550.258927] LNet: 114586:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2550.260520] LNet: 114586:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2551.264380] LNet: Removed LNI 192.168.202.41@tcp [ 2551.270570] LNetError: 114586:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2551.276714] LNetError: 114586:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2551.293445] LNet: Removed LNI 192.168.202.41@tcp2 [ 2551.321078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2551.387488] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 2551.391869] LNet: Added LNI 192.168.202.41@tcp2 [8/256/0/180] [ 2551.399389] LNet: 114592:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2551.782819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.41@tcp [ 2551.944677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2551.962902] LNet: Removed LNI 192.168.202.41@tcp [ 2552.999389] LNet: Removed LNI 192.168.202.41@tcp2 [ 2553.095387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2553.194621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2553.218681] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 2554.176282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2554.193649] LNet: Added LNI 192.168.202.41@tcp2 [8/256/0/180] [ 2555.188806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2555.251907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.41@tcp [ 2555.617376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2555.649418] LNetError: 114908:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2555.656463] LNetError: 114908:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2555.662852] LNet: Removed LNI 192.168.202.41@tcp [ 2555.673627] LNet: Removed LNI 192.168.202.41@tcp2 [ 2555.706903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2555.776530] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 2555.786048] LNet: Added LNI 192.168.202.41@tcp2 [8/256/0/180] [ 2555.793172] LNet: 114914:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2556.533166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.41@tcp [ 2556.843289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2557.920443] LNet: Removed LNI 192.168.202.41@tcp [ 2557.933938] LNet: Removed LNI 192.168.202.41@tcp2 [ 2557.988859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2558.087854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2558.146181] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 2559.224117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2559.287844] LNet: Added LNI 192.168.202.41@tcp2 [8/256/0/180] [ 2560.442832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.41@tcp [ 2560.505849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2560.565912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.41@tcp [ 2560.815853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2560.835367] LNet: 115236:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2560.839703] LNet: 115236:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2561.888151] LNet: Removed LNI 192.168.202.41@tcp [ 2562.919476] LNet: Removed LNI 192.168.202.41@tcp2 [ 2562.993887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2563.102907] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 2563.107895] LNet: Added LNI 192.168.202.41@tcp2 [8/256/0/180] [ 2563.117367] LNet: 115242:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2563.787360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.41@tcp [ 2563.981846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2564.011555] LNetError: 115280:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2564.015316] LNetError: 115280:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2564.022275] LNet: Removed LNI 192.168.202.41@tcp [ 2565.109375] LNet: Removed LNI 192.168.202.41@tcp2 [ 2565.182755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2565.259507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2565.282878] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 2566.024262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2566.041669] LNet: Added LNI 192.168.202.41@tcp2 [8/256/0/180] [ 2566.838660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2566.866355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2566.914112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.41@tcp [ 2567.303686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2568.352569] LNet: Removed LNI 192.168.202.41@tcp [ 2568.366790] LNet: Removed LNI 192.168.202.41@tcp2 [ 2568.416444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2568.516818] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 2568.526048] LNet: Added LNI 192.168.202.41@tcp2 [8/256/0/180] [ 2568.540575] LNet: 115582:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2569.190157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.41@tcp [ 2569.565799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2570.593553] LNet: Removed LNI 192.168.202.41@tcp [ 2570.613526] LNet: Removed LNI 192.168.202.41@tcp2 [ 2570.643468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2570.722933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2570.743940] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 2571.586825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2571.614350] LNet: Added LNI 192.168.202.41@tcp2 [8/256/0/180] [ 2572.658702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.41@tcp [ 2572.745575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2572.781594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2572.855306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.41@tcp [ 2573.292921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2574.369479] LNet: Removed LNI 192.168.202.41@tcp [ 2575.401909] LNet: Removed LNI 192.168.202.41@tcp2 [ 2575.483443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2575.551432] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 2575.555452] LNet: Added LNI 192.168.202.41@tcp2 [8/256/0/180] [ 2576.150804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.41@tcp [ 2585.119051] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2587.108923] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 21:13:43 (1781226823) [ 2587.488909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2587.528185] LNet: 116702:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2587.532976] LNet: 116702:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2587.546045] LNetError: 116702:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2587.554578] LNetError: 116702:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2587.564940] LNet: Removed LNI 192.168.202.41@tcp [ 2588.589027] LNet: Removed LNI 192.168.202.41@tcp2 [ 2588.656885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2589.027645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2589.048061] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 2589.953806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2589.978447] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2591.242245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2591.273908] LNet: Removed LNI 192.168.202.41@tcp [ 2591.281444] LNet: Removed LNI 10.1.2.3@tcp [ 2591.313973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2591.465936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2591.523655] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 2591.530227] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2592.142208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2592.176789] LNet: Removed LNI 192.168.202.41@tcp [ 2592.191807] LNet: Removed LNI 10.1.2.3@tcp [ 2599.775152] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 21:13:55 (1781226835) [ 2600.051850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2600.121321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2600.269550] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2600.508332] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2600.514671] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2600.516742] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2602.938155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2602.979206] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 2603.990625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2604.019344] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2606.145287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.41@tcp [ 2606.178884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2606.239367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.202.41@tcp -m PUT -e local_timeout -r 1 [ 2606.330189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2606.414714] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2606.424419] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2606.850614] LNetError: 117646:0:(lib-move.c:4710:lnet_parse()) 192.168.202.41@tcp, src 192.168.202.41@tcp: Dropping PUT (error -108 looking up sender) [ 2608.864592] LNet: Removed LNI 192.168.202.41@tcp [ 2608.881767] LNet: Removed LNI 10.1.2.3@tcp [ 2609.271303] Key type .llcrypt unregistered [ 2609.284393] Key type ._llcrypt unregistered [ 2623.143749] Key type ._llcrypt registered [ 2623.146964] Key type .llcrypt registered [ 2624.786163] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 21:14:20 (1781226860) [ 2625.365605] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2625.382317] alg: No test for adler32 (adler32-zlib) [ 2626.352173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2626.891280] LNet: 118960:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2627.664236] Key type .llcrypt unregistered [ 2627.672236] Key type ._llcrypt unregistered [ 2634.106379] Key type ._llcrypt registered [ 2634.108373] Key type .llcrypt registered [ 2634.267590] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2634.276176] alg: No test for adler32 (adler32-zlib) [ 2635.244087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2635.297587] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 2644.454746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.141@tcp [ 2644.680775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2645.091305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.141@tcp [ 2645.405893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2645.467136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2645.536264] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.141@tcp [ 2645.546150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2645.649336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2645.732433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2645.796781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2645.865451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2645.971216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2646.014519] LNet: 119410:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2646.056854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2646.144933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2646.226477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2646.273553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2646.315746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2646.355831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2647.115562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2647.381361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2647.434236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2647.474717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2647.553233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2647.619300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2647.686532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2647.738977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2647.830284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2647.872429] LNet: 119528:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2647.909301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2647.969990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2648.003196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2648.047380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2648.150357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2648.263334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2649.443234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.141@tcp [ 2649.824429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2649.869978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2649.920312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2649.950303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2650.007442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2650.056292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2650.132367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2650.213754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2650.252867] LNet: 119645:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2650.269361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2650.291698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2650.323485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2650.367279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2650.437123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2650.480729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2651.731568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2652.264583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2652.321108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2652.402428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2652.488777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2652.562184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2652.632380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2652.709149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2652.764336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2652.819897] LNet: 119763:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2652.855117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2652.969321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2653.058754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2653.111765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2653.255955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2653.312330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2653.942664] LNet: 119809:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2653.946940] LNetError: 119809:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2653.967912] LNet: Removed LNI 192.168.202.41@tcp [ 2654.495628] Key type .llcrypt unregistered [ 2654.497298] Key type ._llcrypt unregistered [ 2676.256620] Key type ._llcrypt registered [ 2676.258844] Key type .llcrypt registered [ 2677.828522] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 21:15:13 (1781226913) [ 2678.187671] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2678.198037] alg: No test for adler32 (adler32-zlib) [ 2679.070198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2679.349778] LNet: 120698:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2679.817408] Key type .llcrypt unregistered [ 2679.824049] Key type ._llcrypt unregistered [ 2686.314278] Key type ._llcrypt registered [ 2686.320917] Key type .llcrypt registered [ 2686.510268] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2686.528278] alg: No test for adler32 (adler32-zlib) [ 2687.502594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2687.543403] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 2696.070611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.141@tcp [ 2708.439873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2708.517616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2708.573801] LNet: Added LNI 192.168.202.41@tcp1 [8/256/0/180] [ 2709.922074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2711.050401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.141@tcp [ 2711.140138] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.141@tcp [ 2711.566176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2711.596791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2711.656406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2711.704820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2711.771490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2711.866448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2711.998517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2712.054479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2712.107696] LNet: 121352:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2712.143245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2712.194315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2712.246523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2712.301932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2712.381136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2712.465993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2713.273960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2713.619856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2713.664604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2713.689185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2713.709870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2713.762752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2713.863596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2713.943794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2713.983895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2714.009012] LNet: 121469:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2714.051946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2714.116326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2714.148229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2714.166643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2714.192301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2714.236589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2715.400833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.141@tcp [ 2715.810814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2715.876163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2715.933980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2716.001516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2716.069987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2716.143447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2716.199228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2716.229832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2716.252840] LNet: 121586:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2716.292913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2716.341823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2716.428699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2716.476922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2716.524336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2716.552395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2717.660799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2718.052758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2718.092863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2718.149808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2718.193368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2718.240448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2718.287958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2718.346476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2718.399952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2718.436267] LNet: 121704:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2718.474471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2718.538222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2718.615208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2718.672405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2718.739731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2718.804670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2724.846162] LNet: 121785:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2724.852572] LNetError: 121785:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2724.859467] LNet: Removed LNI 192.168.202.41@tcp [ 2724.876065] LNet: Removed LNI 192.168.202.41@tcp1 [ 2725.517596] Key type .llcrypt unregistered [ 2725.521767] Key type ._llcrypt unregistered [ 2745.962709] Key type ._llcrypt registered [ 2745.965648] Key type .llcrypt registered [ 2747.753122] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 21:16:23 (1781226983) [ 2756.027575] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 21:16:32 (1781226992) [ 2763.521946] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 21:16:39 (1781226999) [ 2772.370940] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 21:16:48 (1781227008) [ 2780.687655] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 21:16:56 (1781227016) [ 2788.330656] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 21:17:04 (1781227024) [ 2796.857846] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 21:17:12 (1781227032) [ 2804.977472] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 21:17:20 (1781227040) [ 2812.662381] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 21:17:28 (1781227048) [ 2820.732098] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 21:17:36 (1781227056) [ 2828.793516] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 21:17:44 (1781227064) [ 2837.450606] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 21:17:53 (1781227073) [ 2845.664284] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 21:18:01 (1781227081) [ 2853.432057] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 21:18:08 (1781227088) [ 2863.202798] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 21:18:19 (1781227099) [ 2872.316135] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 21:18:28 (1781227108) [ 2881.322632] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 21:18:37 (1781227117) [ 2889.903333] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 21:18:45 (1781227125) [ 2890.522357] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2890.565809] alg: No test for adler32 (adler32-zlib) [ 2891.579874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2891.757960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2891.785418] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 2893.219639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2893.252969] LNet: 132502:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2893.261976] LNetError: 132502:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2893.290646] LNet: Removed LNI 192.168.202.41@tcp [ 2893.305950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2893.380979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2893.421776] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 2901.406537] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 21:18:57 (1781227137) [ 2909.224394] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 21:19:05 (1781227145) [ 2917.605336] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 21:19:13 (1781227153) [ 2927.597699] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 21:19:23 (1781227163) [ 2935.962392] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 21:19:31 (1781227171) [ 2944.967188] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 21:19:40 (1781227180) [ 2953.183799] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 21:19:49 (1781227189) [ 2962.832085] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 21:19:58 (1781227198) [ 2971.434130] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 21:20:07 (1781227207) [ 2980.250401] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 21:20:16 (1781227216) [ 2988.076625] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 21:20:24 (1781227224) [ 2996.546098] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 21:20:32 (1781227232) [ 3005.212351] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2738 sec ========== 21:20:41 (1781227241) [ 3005.541935] LNet: 139969:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3005.547868] LNetError: 139969:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3005.560767] LNet: Removed LNI 192.168.202.41@tcp [ 3006.374720] Key type .llcrypt unregistered [ 3006.376333] Key type ._llcrypt unregistered [ 3020.983895] Key type ._llcrypt registered [ 3020.988421] Key type .llcrypt registered [ 3021.419841] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 3021.434430] alg: No test for adler32 (adler32-zlib) [ 3022.576766] Lustre: Lustre: Build Version: 2.17.53_70_g68e5078 [ 3022.883152] LNet: Added LNI 192.168.202.41@tcp [8/256/0/180] [ 3024.680215] Key type lgssc registered [ 3026.205270] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3094.585578] Lustre: Mounted lustre-client [ 3099.986954] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3108.768220] Lustre: Unmounted lustre-client [ 3147.793292] Key type lgssc unregistered [ 3148.403874] LNet: 142901:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3148.408347] LNetError: 142901:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3148.425480] LNet: Removed LNI 192.168.202.41@tcp [ 3149.975224] Key type .llcrypt unregistered [ 3149.977380] Key type ._llcrypt unregistered