[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 411778821 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002390] x2apic enabled [ 0.003006] Switched APIC routing to physical x2apic. [ 0.004010] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007015] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008007] pid_max: default: 32768 minimum: 301 [ 0.010031] LSM: Security Framework initializing [ 0.012129] Yama: becoming mindful. [ 0.013040] SELinux: Initializing. [ 0.014085] *** VALIDATE selinux *** [ 0.021562] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025673] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026129] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027068] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028085] *** VALIDATE tmpfs *** [ 0.029345] *** VALIDATE proc *** [ 0.030214] *** VALIDATE cgroup *** [ 0.031010] *** VALIDATE cgroup2 *** [ 0.032360] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.033137] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.034003] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.035018] Spectre V2 : User space: Vulnerable [ 0.036003] Speculative Store Bypass: Vulnerable [ 0.038000] debug: unmapping init [mem 0xffffffff9e659000-0xffffffff9e660fff] [ 0.039155] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.041390] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.042042] ... version: 2 [ 0.043022] ... bit width: 48 [ 0.044016] ... generic registers: 4 [ 0.045008] ... value mask: 0000ffffffffffff [ 0.046008] ... max period: 00007fffffffffff [ 0.047008] ... fixed-purpose events: 3 [ 0.048008] ... event mask: 000000070000000f [ 0.049279] rcu: Hierarchical SRCU implementation. [ 0.051372] smp: Bringing up secondary CPUs ... [ 0.052798] x86: Booting SMP configuration: [ 0.053076] .... node #0, CPUs: #1 #2 #3 [ 0.063469] smp: Brought up 1 node, 4 CPUs [ 0.065015] smpboot: Max logical packages: 1 [ 0.066017] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.115130] node 0 deferred pages initialised in 47ms [ 0.119391] devtmpfs: initialized [ 0.120341] x86/mm: Memory block size: 128MB [ 0.123105] gcov: version magic: 0x41383552 [ 0.125399] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.126090] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.127327] pinctrl core: initialized pinctrl subsystem [ 0.128208] [ 0.129008] ************************************************************* [ 0.130010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.131011] ** ** [ 0.132010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.133012] ** ** [ 0.134013] ** This means that this kernel is built to expose internal ** [ 0.135014] ** IOMMU data structures, which may compromise security on ** [ 0.136011] ** your system. ** [ 0.137007] ** ** [ 0.138007] ** If you see this message and you are not debugging the ** [ 0.139008] ** kernel, report this immediately to your vendor! ** [ 0.140011] ** ** [ 0.141012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.142013] ************************************************************* [ 0.143885] NET: Registered protocol family 16 [ 0.144524] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.145080] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.146089] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.147675] cpuidle: using governor menu [ 0.150253] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.154901] PCI: Using configuration type 1 for base access [ 0.157433] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.173314] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.174028] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.176045] cryptd: max_cpu_qlen set to 1000 [ 0.178282] ACPI: Added _OSI(Module Device) [ 0.179000] ACPI: Added _OSI(Processor Device) [ 0.179006] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.179983] ACPI: Added _OSI(Processor Aggregator Device) [ 0.184175] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.191045] ACPI: Interpreter enabled [ 0.192060] ACPI: PM: (supports S0 S3 S4 S5) [ 0.193009] ACPI: Using IOAPIC for interrupt routing [ 0.194116] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.195473] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.207113] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.208063] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.209023] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.210184] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.212354] acpiphp: Slot [2] registered [ 0.213135] acpiphp: Slot [5] registered [ 0.214181] acpiphp: Slot [6] registered [ 0.215189] acpiphp: Slot [3] registered [ 0.216075] acpiphp: Slot [4] registered [ 0.217066] acpiphp: Slot [7] registered [ 0.218048] acpiphp: Slot [8] registered [ 0.219065] acpiphp: Slot [9] registered [ 0.220226] acpiphp: Slot [10] registered [ 0.221081] acpiphp: Slot [11] registered [ 0.222099] acpiphp: Slot [12] registered [ 0.223059] acpiphp: Slot [13] registered [ 0.224122] acpiphp: Slot [14] registered [ 0.225077] acpiphp: Slot [15] registered [ 0.226094] acpiphp: Slot [16] registered [ 0.227161] acpiphp: Slot [17] registered [ 0.228129] acpiphp: Slot [18] registered [ 0.229072] acpiphp: Slot [19] registered [ 0.230146] acpiphp: Slot [20] registered [ 0.231113] acpiphp: Slot [21] registered [ 0.232090] acpiphp: Slot [22] registered [ 0.233119] acpiphp: Slot [23] registered [ 0.234176] acpiphp: Slot [24] registered [ 0.235072] acpiphp: Slot [25] registered [ 0.236075] acpiphp: Slot [26] registered [ 0.237101] acpiphp: Slot [27] registered [ 0.238065] acpiphp: Slot [28] registered [ 0.239108] acpiphp: Slot [29] registered [ 0.240106] acpiphp: Slot [30] registered [ 0.241131] acpiphp: Slot [31] registered [ 0.242057] PCI host bridge to bus 0000:00 [ 0.243018] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.244022] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.245029] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.246139] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.247019] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.248031] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.249205] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.251299] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.253568] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.257015] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.259330] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.260019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.261024] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.262017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.263823] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.265122] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.266038] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.268083] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.270826] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.277017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.279791] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.289000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.293017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.296021] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.304017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.309409] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.317020] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.325019] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.351028] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.362000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.362407] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.363467] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.364000] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.364000] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.367156] iommu: Default domain type: Passthrough [ 0.369756] SCSI subsystem initialized [ 0.371164] ACPI: bus type USB registered [ 0.372088] usbcore: registered new interface driver usbfs [ 0.374056] usbcore: registered new interface driver hub [ 0.376059] usbcore: registered new device driver usb [ 0.377249] pps_core: LinuxPPS API ver. 1 registered [ 0.379007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.381057] PTP clock support registered [ 0.383307] EDAC MC: Ver: 3.0.0 [ 0.387083] PCI: Using ACPI for IRQ routing [ 0.388997] NetLabel: Initializing [ 0.401014] NetLabel: domain hash size = 128 [ 0.403012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.409133] NetLabel: unlabeled traffic allowed by default [ 0.411122] vgaarb: loaded [ 0.412233] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.417016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.430284] clocksource: Switched to clocksource kvm-clock [ 0.570803] VFS: Disk quotas dquot_6.6.0 [ 0.575963] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.589793] *** VALIDATE ramfs *** [ 0.592339] *** VALIDATE hugetlbfs *** [ 0.594407] pnp: PnP ACPI init [ 0.596898] pnp: PnP ACPI: found 6 devices [ 0.636529] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.640705] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.643571] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.646413] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.649447] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.652076] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.655265] NET: Registered protocol family 2 [ 0.657958] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.663959] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.670538] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.689049] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.693514] TCP: Hash tables configured (established 65536 bind 65536) [ 0.703676] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.711594] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.723670] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.728204] NET: Registered protocol family 1 [ 0.730939] RPC: Registered named UNIX socket transport module. [ 0.736104] RPC: Registered udp transport module. [ 0.738376] RPC: Registered tcp transport module. [ 0.740410] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.743651] NET: Registered protocol family 44 [ 0.745424] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.748450] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.750739] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.753233] PCI: CLS 0 bytes, default 64 [ 0.755315] Unpacking initramfs... [ 2.681476] debug: unmapping init [mem 0xffff927b7cc64000-0xffff927b7ffcffff] [ 2.687815] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.689760] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.698330] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.592624] Initialise system trusted keyrings [ 3.594300] Key type blacklist registered [ 3.598372] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.609814] zbud: loaded [ 3.612476] *** VALIDATE nfs *** [ 3.613800] *** VALIDATE nfs4 *** [ 3.615270] pstore: using deflate compression [ 3.619514] Platform Keyring initialized [ 3.719334] NET: Registered protocol family 38 [ 3.720806] Key type asymmetric registered [ 3.725078] Asymmetric key parser 'x509' registered [ 3.727371] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.729891] io scheduler mq-deadline registered [ 3.731438] io scheduler kyber registered [ 3.732823] io scheduler bfq registered [ 3.738706] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.741468] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.744794] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.747547] ACPI: Power Button [PWRF] [ 3.753084] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.761155] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.772223] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.805800] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.839458] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.843671] Non-volatile memory driver v1.3 [ 3.846165] Linux agpgart interface v0.103 [ 3.884821] virtio_blk virtio1: [vda] 145800 512-byte logical blocks (74.6 MB/71.2 MiB) [ 3.887354] vda: detected capacity change from 0 to 74649600 [ 3.920556] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.923352] vdb: detected capacity change from 0 to 1073741824 [ 3.937295] libphy: Fixed MDIO Bus: probed [ 3.972506] usbcore: registered new interface driver usbserial_generic [ 3.975465] usbserial: USB Serial support registered for generic [ 3.982055] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.987580] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.989507] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.991490] mousedev: PS/2 mouse device common for all mice [ 3.994898] rtc_cmos 00:05: RTC can wake from S4 [ 3.998975] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.004846] rtc_cmos 00:05: registered as rtc0 [ 4.006692] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.010281] intel_pstate: CPU model not supported [ 4.015553] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.016087] hid: raw HID events driver (C) Jiri Kosina [ 4.023152] usbcore: registered new interface driver usbhid [ 4.025155] usbhid: USB HID core driver [ 4.026877] drop_monitor: Initializing network drop monitor service [ 4.029346] Initializing XFRM netlink socket [ 4.030749] NET: Registered protocol family 10 [ 4.033112] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.037805] Segment Routing with IPv6 [ 4.040440] NET: Registered protocol family 17 [ 4.042321] mpls_gso: MPLS GSO support [ 4.048219] RAS: Correctable Errors collector initialized. [ 4.050413] AVX version of gcm_enc/dec engaged. [ 4.052181] AES CTR mode by8 optimization enabled [ 4.125387] sched_clock: Marking stable (4125351661, 0)->(5281292450, -1155940789) [ 4.128556] registered taskstats version 1 [ 4.130791] Loading compiled-in X.509 certificates [ 4.133382] zswap: loaded using pool lzo/zbud [ 4.155568] Key type big_key registered [ 4.168247] Key type encrypted registered [ 4.169896] ima: No TPM chip found, activating TPM-bypass! [ 4.172095] ima: Allocated hash algorithm: sha1 [ 4.173699] ima: No architecture policies found [ 4.175567] evm: Initialising EVM extended attributes: [ 4.177322] evm: security.selinux [ 4.178551] evm: security.ima [ 4.179568] evm: security.capability [ 4.180785] evm: HMAC attrs: 0x1 [ 4.183535] rtc_cmos 00:05: setting system clock to 2026-07-17 11:37:43 UTC (1784288263) [ 4.191610] debug: unmapping init [mem 0xffffffff9f603000-0xffffffff9f7fffff] [ 4.195275] debug: unmapping init [mem 0xffffffff9e382000-0xffffffff9e658fff] [ 4.207889] Write protecting the kernel read-only data: 28672k [ 4.212944] debug: unmapping init [mem 0xffffffff9ca03000-0xffffffff9cbfffff] [ 4.217820] debug: unmapping init [mem 0xffffffff9d314000-0xffffffff9d3fffff] [ 4.250580] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.258607] systemd[1]: Detected virtualization kvm. [ 4.260566] systemd[1]: Detected architecture x86-64. [ 4.262542] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.290512] systemd[1]: No hostname configured. [ 4.292250] systemd[1]: Set hostname to . [ 4.294319] random: systemd: uninitialized urandom read (16 bytes read) [ 4.297062] systemd[1]: Initializing machine ID from random generator. [ 4.385352] random: ln: uninitialized urandom read (6 bytes read) [ 4.506499] random: systemd: uninitialized urandom read (16 bytes read) [ 4.508209] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 4.513622] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 4.526403] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Timers. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket. Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Initrd Root Device. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Journal Service. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.574450] device-mapper: uevent: version 1.0.3 [ 5.576403] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 6.772556] virtio_net virtio0 ens2: renamed from eth0 [ 7.081263] scsi host0: ata_piix [ 7.086913] scsi host1: ata_piix [ 7.089432] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 7.096591] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.883123] random: fast init done [ 12.698539] random: crng init done [ 12.701760] random: 7 urandom warning(s) missed due to ratelimiting [ 14.632329] dracut-initqueue[591]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 15.594366] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 17.592951] printk: systemd: 19 output lines suppressed due to ratelimiting [ 18.309865] SELinux: Disabled at runtime. [ 18.445670] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 18.463520] systemd[1]: Detected virtualization kvm. [ 18.467019] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 19.530827] systemd[1]: initrd-switch-root.service: Succeeded. [ 19.534781] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 19.547098] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 19.551973] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 19.560256] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 19.577705] systemd[1]: Starting Journal Service... Starting Journal Service... [ 19.587960] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Switch Root. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice User and Session Slice. [ OK ] Listening on RPCbind Server Activation Socket. Mounting Huge Pages File System... [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Activating swap /dev/disk/by-label/SWAP... Mounting POSIX Message Queue File System... [ OK ] Created slice system-getty.slice. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on initctl Compatibility Named Pipe. [ 19.817138] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Apply Kernel Variables... Starting udev Coldplug all Devices... [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target Paths. Mounting Kernel Debug File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 20.494823] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 21.341260] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 21.408828] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 22.088104] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 22.134757] EDAC sbridge: Ver: 1.1.2 [ 25.250791] Key type dns_resolver registered [ 25.982499] NFS: Registering the id_resolver key type [ 25.988262] Key type id_resolver registered [ 25.992115] Key type id_legacy registered [* ] A start job is running for Configur…-only root support (6s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Authorization Manager. [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg113-client login: [ 56.976138] libcfs: loading out-of-tree module taints kernel. [ 57.010886] Key type ._llcrypt registered [ 57.012010] Key type .llcrypt registered [ 57.272595] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 57.278301] alg: No test for adler32 (adler32-zlib) [ 58.239985] Lustre: Lustre: Build Version: 2.17.54_173_g0a24909 [ 58.518297] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 60.127151] Key type lgssc registered [ 60.679237] Lustre: Echo OBD driver; http://www.lustre.org/ [ 108.815417] Lustre: Mounted lustre-client [ 111.161679] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 116.897483] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing check_logdir /tmp/testlogs/ [ 118.611016] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing yml_node [ 120.316227] Lustre: DEBUG MARKER: Client: 2.17.54.173 [ 121.305210] Lustre: DEBUG MARKER: MDS: 2.17.54.173 [ 122.372532] Lustre: DEBUG MARKER: OSS: 2.17.54.173 [ 123.000685] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Fri Jul 17 07:39:41 EDT 2026 [ 128.176697] Lustre: Unmounted lustre-client [ 153.248314] Key type lgssc unregistered [ 153.358433] LNet: 8373:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 153.360871] LNetError: 8373:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 153.368587] LNet: Removed LNI 192.168.201.13@tcp [ 153.648149] Key type .llcrypt unregistered [ 153.649615] Key type ._llcrypt unregistered [ 161.992551] Key type ._llcrypt registered [ 161.994215] Key type .llcrypt registered [ 162.054341] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 162.061108] alg: No test for adler32 (adler32-zlib) [ 162.862495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 162.890583] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 162.903716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 164.197289] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 164.222080] LNet: 9247:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 164.225195] LNetError: 9247:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 164.231859] LNet: Removed LNI 192.168.201.13@tcp [ 164.432612] Key type .llcrypt unregistered [ 164.433757] Key type ._llcrypt unregistered [ 166.228884] Key type ._llcrypt registered [ 166.230389] Key type .llcrypt registered [ 166.646544] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 07:40:25 (1784288425) [ 166.749639] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 166.754396] alg: No test for adler32 (adler32-zlib) [ 167.552288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 167.579661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 167.589036] LNet: 9663:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 167.596150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 167.612584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 167.619040] LNet: 9670:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 169.760511] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 07:40:28 (1784288428) [ 169.795502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 169.805703] LNet: 10244:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 169.818432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 169.847168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 171.845250] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 07:40:30 (1784288430) [ 171.883646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 171.897451] LNet: 10826:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 171.909795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 171.941727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 173.950832] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 07:40:32 (1784288432) [ 173.977116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 173.986817] LNet: 11408:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 173.997334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 174.025216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 176.124506] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 07:40:34 (1784288434) [ 176.149711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 176.167625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 176.203064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 176.335809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 178.586116] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 07:40:37 (1784288437) [ 178.629620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 178.641039] LNet: 12584:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 178.644931] LNet: 12584:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 178.656512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 178.691199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 178.864566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 179.031031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 181.213387] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 07:40:40 (1784288440) [ 181.247845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 181.263660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 181.291411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 183.452404] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 07:40:42 (1784288442) [ 183.480052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 183.496094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 183.520840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 183.533701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 183.666286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 183.682053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 183.823954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 183.839771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 183.984586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 183.997388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 184.125339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 184.138390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 184.268448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 184.281988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 184.404926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 184.416116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 184.550424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 184.562526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 184.689169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 184.702828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 184.834522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 184.846903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 184.981440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 184.994640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 185.115316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 185.131214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 185.252623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 185.266678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 187.311266] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 07:40:46 (1784288446) [ 187.337858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 187.346953] LNet: 14442:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 187.349432] LNet: 14442:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 187.356914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 187.385535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 187.398901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 189.515458] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 07:40:48 (1784288448) [ 189.560206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 189.577079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 189.608536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 189.622511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 191.672405] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 07:40:50 (1784288450) [ 191.698963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 191.715175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 191.748803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 191.763992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 193.749293] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 07:40:52 (1784288452) [ 193.796750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 193.812791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 193.848495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 193.861701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 195.900198] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 07:40:54 (1784288454) [ 195.927992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 195.944486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 195.975708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 195.990050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 198.018246] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 07:40:56 (1784288456) [ 198.044528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 198.061242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 198.092667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 198.107660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 200.233747] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 07:40:59 (1784288459) [ 200.264772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 200.281412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 200.317311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 200.467088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 202.684827] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 07:41:01 (1784288461) [ 202.714457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 202.730735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 202.807665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 202.885709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 204.965528] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 07:41:03 (1784288463) [ 204.996689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 205.004980] LNet: 19169:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 205.008209] LNet: 19169:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 205.020117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 205.064246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 205.136481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 207.243478] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 07:41:06 (1784288466) [ 207.283628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 207.298183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 207.457248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 207.546157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 209.538065] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 07:41:08 (1784288468) [ 209.568251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 209.583718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 209.610868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 209.627656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 211.630608] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 07:41:10 (1784288470) [ 211.666527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 211.683531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 211.718343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 211.738903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 213.858878] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 07:41:12 (1784288472) [ 213.910629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 213.927630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 213.955455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 215.884550] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 07:41:14 (1784288474) [ 215.912671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 215.928522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 215.956569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 218.082935] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 07:41:16 (1784288476) [ 218.111767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 218.134539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 218.173815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 218.310493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 220.556173] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 07:41:19 (1784288479) [ 220.592528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 220.612354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 220.651594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 220.668475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 222.998171] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 07:41:21 (1784288481) [ 223.042146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 223.061058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 223.091541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 223.106600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 225.251708] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 07:41:24 (1784288484) [ 225.291493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 225.309249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 225.341156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 225.356719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 227.564618] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 07:41:26 (1784288486) [ 227.600852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 227.617733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 227.651722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 227.668331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 229.784180] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 07:41:28 (1784288488) [ 229.822171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 229.835628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 229.860679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 229.886958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 229.893675] LNetError: 25747:0:(peer.c:2276:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 229.909315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 229.921927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 231.850870] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 07:41:30 (1784288490) [ 231.880224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 231.898217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 231.935086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.xb5fDT0cbE [ 231.961463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.xb5fDT0cbE [ 231.985714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.xb5fDT0cbE [ 233.904101] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 07:41:32 (1784288492) [ 233.927856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 233.946195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 233.975220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 233.990609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 234.010327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 235.962325] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 07:41:34 (1784288494) [ 235.991436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 236.005949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 236.030090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 236.038486] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 236.483768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.13@tcp [ 236.891575] Lustre: Lustre: Build Version: 2.17.54_173_g0a24909 [ 238.567168] Key type lgssc registered [ 238.917719] Lustre: Echo OBD driver; http://www.lustre.org/ [ 240.025474] LNetError: 27506:0:(lib-move.c:2120:lnet_handle_find_routed_path()) peer 192.168.201.13@tcp99 has no available nets [ 240.918304] Key type lgssc unregistered [ 241.080485] LNet: 28612:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 241.082820] LNet: 28612:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 14 previous similar messages [ 241.085094] LNetError: 28612:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 241.096244] LNet: Removed LNI 192.168.201.13@tcp [ 241.305942] Key type .llcrypt unregistered [ 241.307211] Key type ._llcrypt unregistered [ 243.208971] Key type ._llcrypt registered [ 243.210179] Key type .llcrypt registered [ 243.644709] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 07:41:42 (1784288502) [ 243.757134] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 243.762729] alg: No test for adler32 (adler32-zlib) [ 244.562378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 244.585880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 245.567660] LNet: Message forwarding will be enabled in 2s [ 248.671119] LNet: Message forwarding enabled [ 248.715345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 248.722369] LNet: Message forwarding will stop in 2s to 6s [ 252.872603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 253.459437] LNet: Message forwarding disabled [ 253.853741] LNet: Message forwarding will be enabled in 2s [ 256.022663] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 07:41:54 (1784288514) [ 256.055332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 256.065692] LNet: 30012:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 256.596814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 256.624666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 258.564227] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 07:41:57 (1784288517) [ 258.592232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 258.600723] LNet: 30591:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 258.610721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 258.638497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 258.652354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 258.665974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 258.679179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 258.691272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 258.702973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 258.715433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 258.728597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 258.740958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 258.751657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 258.763703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 258.778079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 258.792618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 258.806030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 258.818814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 258.833735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 258.848605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 258.863544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 258.875595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 258.883289] LNetError: 30640:0:(router.c:624:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 261.068512] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 07:41:59 (1784288519) [ 261.110826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 261.123770] LNet: 31207:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 261.139594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 261.174386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 263.393159] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 07:42:02 (1784288522) [ 263.989397] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg113-server [ 264.609264] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 07:42:03 (1784288523) [ 264.637559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 264.647698] LNet: 32134:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 264.660941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 264.687729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 264.698962] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 265.008659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.201.[14-18]@tcp [ 265.017792] LNet: 32273:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 265.029140] LNetError: 32144:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.16@tcp is being used as a gateway but routing feature is not turned on [ 265.039048] LNetError: 32144:0:(lib-lnet.h:1272:lnet_set_route_aliveness()) route to tcp1 through 192.168.201.16@tcp has gone from up to down [ 266.084747] LNetError: 32144:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.17@tcp is being used as a gateway but routing feature is not turned on [ 266.093479] LNetError: 32144:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 3 previous similar messages [ 267.107139] LNetError: 32144:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.17@tcp is being used as a gateway but routing feature is not turned on [ 267.112372] LNetError: 32144:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 1 previous similar message [ 267.291050] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 07:42:06 (1784288526) [ 267.936146] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg113-server [ 268.127339] LNetError: 32150:0:(lib-lnet.h:1272:lnet_set_route_aliveness()) route to tcp1 through 192.168.201.18@tcp has gone from up to down [ 268.137458] LNetError: 32150:0:(lib-lnet.h:1272:lnet_set_route_aliveness()) Skipped 1 previous similar message [ 268.570757] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 07:42:07 (1784288527) [ 268.625336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 268.636830] LNetError: 33207:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 268.647958] LNet: Removed LNI 192.168.201.13@tcp [ 268.656845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 268.676532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 268.687094] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 268.954057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.201.[14-19/2]@tcp [ 268.965394] LNet: 33325:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 268.969670] LNet: 33325:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 268.977932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.201.[14-19/2]@tcp [ 271.239770] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 07:42:10 (1784288530) [ 271.277335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 271.287717] LNet: 33913:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 271.291161] LNet: 33913:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 271.293627] LNetError: 33913:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 271.301871] LNet: Removed LNI 192.168.201.13@tcp [ 271.310441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 271.344071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 271.366858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 271.380626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 271.405746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 271.426314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 271.438236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 271.458536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 271.486291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 271.505359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 271.533647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 271.565683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 271.578958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 271.601164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 271.637821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 271.654731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 271.682934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 271.712866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 271.727666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 271.748663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 271.775607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 271.797918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 271.827029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 271.857589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 271.871388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 271.894286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 271.924949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 271.941298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 271.966687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 271.995534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 272.010390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 272.033295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 272.057491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 272.070711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 272.101539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 274.262188] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 07:42:13 (1784288533) [ 274.957864] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg113-server [ 275.560102] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 07:42:14 (1784288534) [ 276.226535] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg113-server [ 276.859548] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 07:42:15 (1784288535) [ 277.160589] Key type .llcrypt unregistered [ 277.161867] Key type ._llcrypt unregistered [ 279.221152] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 279.283376] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 279.285644] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 279.287965] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 281.524815] Key type ._llcrypt registered [ 281.525727] Key type .llcrypt registered [ 281.592761] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 281.598562] alg: No test for adler32 (adler32-zlib) [ 282.404018] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 282.425360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 282.435949] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 282.674267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 282.683267] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 283.352588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 283.360121] LNet: Removed LNI 10.1.2.3@tcp [ 283.396568] LNet: 36037:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 283.399360] LNetError: 36037:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 283.410329] LNet: Removed LNI 192.168.201.13@tcp [ 283.616399] Key type .llcrypt unregistered [ 283.618082] Key type ._llcrypt unregistered [ 287.076719] Key type ._llcrypt registered [ 287.078425] Key type .llcrypt registered [ 287.523228] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 07:42:26 (1784288546) [ 287.697292] Key type .llcrypt unregistered [ 287.698793] Key type ._llcrypt unregistered [ 289.760889] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 289.802843] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 289.804615] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 289.806852] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 292.049313] Key type ._llcrypt registered [ 292.050418] Key type .llcrypt registered [ 292.127227] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 292.134348] alg: No test for adler32 (adler32-zlib) [ 292.950776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 292.974353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 292.985381] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 293.231398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 293.242515] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 294.129080] LNet: 37521:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 294.131673] LNetError: 37521:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 294.134565] LNet: Removed LNI 10.1.2.3@tcp [ 294.141276] LNet: Removed LNI 192.168.201.13@tcp [ 294.377371] Key type .llcrypt unregistered [ 294.378588] Key type ._llcrypt unregistered [ 298.481484] Key type ._llcrypt registered [ 298.483244] Key type .llcrypt registered [ 298.985996] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 07:42:37 (1784288557) [ 299.154241] Key type .llcrypt unregistered [ 299.156382] Key type ._llcrypt unregistered [ 301.368425] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 301.436223] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 301.438416] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 301.440960] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 303.751736] Key type ._llcrypt registered [ 303.753178] Key type .llcrypt registered [ 303.830629] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 303.836962] alg: No test for adler32 (adler32-zlib) [ 304.650510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 304.676180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 304.693328] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 305.429817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 305.439728] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 306.149659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 306.158139] LNet: Removed LNI 10.1.2.3@tcp [ 306.164212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 306.180509] LNet: Removed LNI 10.1.2.31@tcp [ 306.187155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 306.196667] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 306.447988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 306.456519] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 306.803587] LNet: 39432:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 306.807122] LNetError: 39432:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 306.810397] LNet: Removed LNI 10.1.2.31@tcp [ 306.818661] LNet: Removed LNI 10.1.2.3@tcp [ 307.048352] Key type .llcrypt unregistered [ 307.049903] Key type ._llcrypt unregistered [ 311.012562] Key type ._llcrypt registered [ 311.014166] Key type .llcrypt registered [ 311.591408] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 07:42:50 (1784288570) [ 311.725380] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 311.731496] alg: No test for adler32 (adler32-zlib) [ 312.553410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 312.584929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 312.599376] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 312.940190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 312.948441] LNet: 40383:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 312.950791] LNetError: 40383:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 312.967338] LNet: Removed LNI 192.168.201.13@tcp [ 312.973924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 312.995960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.201.13@tcp [ 313.007921] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 313.287857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.201.13@tcp [ 313.299529] LNet: Removed LNI 192.168.201.13@tcp [ 313.541364] Key type .llcrypt unregistered [ 313.543174] Key type ._llcrypt unregistered [ 317.253561] Key type ._llcrypt registered [ 317.255149] Key type .llcrypt registered [ 317.766982] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 07:42:56 (1784288576) [ 318.446479] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg113-server [ 319.126579] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 07:42:57 (1784288577) [ 319.305514] Key type .llcrypt unregistered [ 319.306906] Key type ._llcrypt unregistered [ 321.410728] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 321.470228] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 321.473944] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 321.476571] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 323.662725] Key type ._llcrypt registered [ 323.663669] Key type .llcrypt registered [ 323.727556] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 323.733982] alg: No test for adler32 (adler32-zlib) [ 324.541136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 324.567042] LNet: Added LNI 192.168.201.13@tcp1 [8/256/0/180] [ 324.829235] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 325.569862] LNet: 42345:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 325.573889] LNetError: 42345:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 325.577106] LNet: Removed LNI 192.168.201.13@tcp1 [ 325.582471] LNet: Removed LNI 10.1.2.3@tcp2 [ 325.806755] Key type .llcrypt unregistered [ 325.807985] Key type ._llcrypt unregistered [ 329.377911] Key type ._llcrypt registered [ 329.379106] Key type .llcrypt registered [ 329.795755] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 07:43:08 (1784288588) [ 329.907737] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 329.913514] alg: No test for adler32 (adler32-zlib) [ 330.711807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 330.730454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 330.741512] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 331.001680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 331.009560] LNet: 43311:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 331.011820] LNetError: 43311:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 331.026316] LNet: Removed LNI 192.168.201.13@tcp [ 331.036743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 331.058887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 331.068178] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 333.145880] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 07:43:12 (1784288592) [ 333.175234] LNet: 43895:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 333.177547] LNetError: 43895:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 333.186644] LNet: Removed LNI 192.168.201.13@tcp [ 333.403772] Key type .llcrypt unregistered [ 333.405101] Key type ._llcrypt unregistered [ 335.254372] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 335.305038] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 335.307462] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 335.310388] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 337.516666] Key type ._llcrypt registered [ 337.517662] Key type .llcrypt registered [ 337.583540] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 337.589813] alg: No test for adler32 (adler32-zlib) [ 338.395071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 338.416072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 338.426189] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 338.666780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 338.674259] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 339.382364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 339.391063] LNet: 44645:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 339.393965] LNetError: 44645:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 339.396497] LNet: Removed LNI 10.1.2.3@tcp [ 339.406349] LNet: Removed LNI 192.168.201.13@tcp [ 339.413275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 339.437866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 339.448298] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 339.450756] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 339.595606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 339.603990] LNet: Removed LNI 10.1.2.3@tcp [ 339.617363] LNet: Removed LNI 192.168.201.13@tcp [ 339.625697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 339.651664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 339.662478] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 339.664193] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 339.809120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 339.823947] LNet: Added LNI 192.168.201.13@tcp2 [8/256/0/180] [ 340.165146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 340.173701] LNet: 44834:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 340.176135] LNet: 44834:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 340.178648] LNetError: 44834:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 340.181849] LNetError: 44834:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 340.184815] LNet: Removed LNI 10.1.2.3@tcp [ 340.185899] LNet: Removed LNI 192.168.201.13@tcp [ 340.194372] LNet: Removed LNI 192.168.201.13@tcp2 [ 340.202741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 340.228730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 340.239663] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 340.241916] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 340.244568] LNet: Added LNI 192.168.201.13@tcp2 [8/256/0/180] [ 342.701259] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 07:43:21 (1784288601) [ 342.743533] LNet: 45425:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 342.752362] LNetError: 45425:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 342.759748] LNetError: 45425:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 342.766356] LNet: Removed LNI 10.1.2.3@tcp [ 342.767776] LNet: Removed LNI 192.168.201.13@tcp [ 342.773638] LNet: Removed LNI 192.168.201.13@tcp2 [ 343.021715] Key type .llcrypt unregistered [ 343.023076] Key type ._llcrypt unregistered [ 345.198327] Key type ._llcrypt registered [ 345.199733] Key type .llcrypt registered [ 345.274991] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 345.280989] alg: No test for adler32 (adler32-zlib) [ 346.094115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 346.115616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 346.124568] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 346.391432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 346.399542] LNet: 45825:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 346.401693] LNetError: 45825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 346.408316] LNet: Removed LNI 192.168.201.13@tcp [ 346.414979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 346.436628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 346.445501] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 346.579541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 346.593262] LNet: Removed LNI 192.168.201.13@tcp [ 346.599872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 346.619929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 346.629567] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 346.768571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 346.791323] LNet: Removed LNI 192.168.201.13@tcp [ 346.800097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 346.819441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 346.828591] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 346.971167] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 347.024801] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 347.026885] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 347.028581] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 349.160708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 349.169539] LNet: 45964:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 349.172535] LNet: 45964:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 349.175625] LNetError: 45964:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 349.178719] LNetError: 45964:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 349.186284] LNet: Removed LNI 192.168.201.13@tcp [ 349.196615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 349.224475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 349.232959] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 351.470987] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 07:43:30 (1784288610) [ 351.504567] LNet: 46553:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 351.510714] LNetError: 46553:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 351.519773] LNet: Removed LNI 192.168.201.13@tcp [ 351.755768] Key type .llcrypt unregistered [ 351.757420] Key type ._llcrypt unregistered [ 353.707074] Key type ._llcrypt registered [ 353.708506] Key type .llcrypt registered [ 353.774233] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 353.780105] alg: No test for adler32 (adler32-zlib) [ 354.577192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 354.594633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 354.603101] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 354.874707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 354.882962] LNet: 46960:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 354.886385] LNetError: 46960:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 354.896337] LNet: Removed LNI 192.168.201.13@tcp [ 354.905360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 354.927622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 354.937568] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 356.904062] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 07:43:35 (1784288615) [ 356.935959] LNet: 47545:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 356.938401] LNetError: 47545:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 356.948633] LNet: Removed LNI 192.168.201.13@tcp [ 357.150484] Key type .llcrypt unregistered [ 357.151620] Key type ._llcrypt unregistered [ 358.987210] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 359.025199] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 359.027146] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 359.028963] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 361.222370] Key type ._llcrypt registered [ 361.223762] Key type .llcrypt registered [ 361.285924] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 361.291158] alg: No test for adler32 (adler32-zlib) [ 362.093925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 362.113627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 362.122462] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 362.411869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 362.419854] LNet: 48000:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 362.422193] LNetError: 48000:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 362.434262] LNet: Removed LNI 192.168.201.13@tcp [ 362.441835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 362.462406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 362.473237] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 364.544247] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 07:43:43 (1784288623) [ 364.574251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 364.583285] LNet: 48593:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 364.586340] LNetError: 48593:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 364.593486] LNet: Removed LNI 192.168.201.13@tcp [ 364.603659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 364.643317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 366.501661] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 07:43:45 (1784288625) [ 366.530354] LNet: 49172:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 366.745482] Key type .llcrypt unregistered [ 366.746544] Key type ._llcrypt unregistered [ 368.551205] Key type ._llcrypt registered [ 368.552279] Key type .llcrypt registered [ 368.620879] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 368.625758] alg: No test for adler32 (adler32-zlib) [ 369.426347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 369.446954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 369.456625] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 369.756986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 369.765717] LNet: 49575:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 369.768235] LNetError: 49575:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 369.784281] LNet: Removed LNI 192.168.201.13@tcp [ 369.792292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 369.817162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 369.826698] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 371.970417] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 07:43:50 (1784288630) [ 372.025350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 372.034994] LNet: 50169:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 372.037927] LNetError: 50169:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 372.046756] LNet: Removed LNI 192.168.201.13@tcp [ 372.055054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 372.084748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 374.006173] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 07:43:52 (1784288632) [ 374.496779] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 375.038203] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 07:43:53 (1784288633) [ 375.558919] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 376.116285] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 07:43:54 (1784288634) [ 376.618284] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 377.161299] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 07:43:56 (1784288636) [ 377.688471] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 378.258171] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 07:43:57 (1784288637) [ 378.763981] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 379.293335] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 07:43:58 (1784288638) [ 379.323826] LNet: 52210:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 379.541539] Key type .llcrypt unregistered [ 379.542856] Key type ._llcrypt unregistered [ 381.401291] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 381.463219] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 381.465808] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 381.468584] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 383.643617] Key type ._llcrypt registered [ 383.645118] Key type .llcrypt registered [ 383.699179] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 383.704416] alg: No test for adler32 (adler32-zlib) [ 384.508594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 384.532931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 384.540405] LNet: Added LNI 192.168.201.13@tcp [8/512/0/180] [ 384.542077] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 384.543442] LNet: 52538:0:(api-ni.c:2931:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 384.573341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 384.581785] LNet: 52557:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 384.585233] LNetError: 52557:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 384.587993] LNet: Removed LNI 10.1.2.3@tcp [ 384.589053] LNetError: 52557:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 384.598402] LNet: Removed LNI 192.168.201.13@tcp [ 384.606779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 384.634857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 384.645976] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 384.670657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 384.679085] LNet: 52581:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 384.682847] LNetError: 52581:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 384.692360] LNet: Removed LNI 192.168.201.13@tcp [ 384.700974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 384.725082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 384.733343] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 384.735233] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 384.736651] LNet: 52589:0:(api-ni.c:2931:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 384.772043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 384.780397] LNet: 52607:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 384.783784] LNetError: 52607:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 384.786408] LNet: Removed LNI 10.1.2.3@tcp [ 384.788249] LNetError: 52607:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 384.799335] LNet: Removed LNI 192.168.201.13@tcp [ 384.808087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 384.833739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 384.843845] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 384.846120] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 384.847728] LNet: 52615:0:(api-ni.c:2931:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 384.912350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 384.920976] LNet: 52641:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 384.923425] LNetError: 52641:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 384.925359] LNet: Removed LNI 10.1.2.3@tcp [ 384.926192] LNetError: 52641:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 384.932314] LNet: Removed LNI 192.168.201.13@tcp [ 384.939405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 384.966960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 384.978723] LNet: Added LNI 192.168.201.13@tcp [8/256/0/182] [ 384.981269] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 385.019876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 385.028460] LNet: 52669:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 385.031227] LNetError: 52669:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 385.033638] LNet: Removed LNI 10.1.2.3@tcp [ 385.034491] LNetError: 52669:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 385.042405] LNet: Removed LNI 192.168.201.13@tcp [ 385.051215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 385.074127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 385.083351] LNet: Added LNI 192.168.201.13@tcp [8/256/0/182] [ 385.085477] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 385.086635] LNet: 52677:0:(api-ni.c:2931:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 385.115549] LNet: Removed LNI 10.1.2.3@tcp [ 385.128318] LNet: Removed LNI 192.168.201.13@tcp [ 385.344917] Key type .llcrypt unregistered [ 385.346208] Key type ._llcrypt unregistered [ 388.845178] Key type ._llcrypt registered [ 388.846193] Key type .llcrypt registered [ 389.298993] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 07:44:08 (1784288648) [ 389.414142] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 389.420286] alg: No test for adler32 (adler32-zlib) [ 390.221367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 390.244213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 390.255433] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 390.507946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.201.13 [ 390.516433] LNet: 53666:0:(config.c:1739:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 390.520607] LNet: Removed LNI 192.168.201.13@tcp [ 392.456227] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 07:44:11 (1784288651) [ 392.482910] LNet: 54228:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 392.485124] LNetError: 54228:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 392.492529] LNet: Removed LNI 192.168.201.13@tcp [ 392.702459] Key type .llcrypt unregistered [ 392.703548] Key type ._llcrypt unregistered [ 394.666022] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 394.762263] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 397.059316] Key type ._llcrypt registered [ 397.060588] Key type .llcrypt registered [ 397.114190] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 397.120173] alg: No test for adler32 (adler32-zlib) [ 398.083750] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 400.187725] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 07:44:19 (1784288659) [ 400.221181] LNet: 55141:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 400.225604] LNetError: 55141:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 400.238682] LNet: Removed LNI 10.1.2.3@tcp [ 400.487478] Key type .llcrypt unregistered [ 400.488796] Key type ._llcrypt unregistered [ 402.394200] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 402.483641] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 402.485339] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 402.486925] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 404.745764] Key type ._llcrypt registered [ 404.746973] Key type .llcrypt registered [ 404.808944] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 404.815436] alg: No test for adler32 (adler32-zlib) [ 405.632176] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 407.799568] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 07:44:26 (1784288666) [ 408.266120] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 408.821244] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 07:44:27 (1784288667) [ 408.849478] LNet: 56324:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 408.852982] LNetError: 56324:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 408.861702] LNet: Removed LNI 10.1.2.3@tcp [ 409.098526] Key type .llcrypt unregistered [ 409.100272] Key type ._llcrypt unregistered [ 410.997675] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 411.089419] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 411.091209] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 411.093905] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 413.364801] Key type ._llcrypt registered [ 413.366063] Key type .llcrypt registered [ 413.424653] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 413.430042] alg: No test for adler32 (adler32-zlib) [ 414.235462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 414.263341] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 417.002866] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 07:44:35 (1784288675) [ 417.131647] LNet: 57558:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 417.133975] LNetError: 57558:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 417.143546] LNet: Removed LNI 10.1.2.3@tcp [ 417.363345] Key type .llcrypt unregistered [ 417.364614] Key type ._llcrypt unregistered [ 419.220036] Key type ._llcrypt registered [ 419.221240] Key type .llcrypt registered [ 419.289371] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 419.294597] alg: No test for adler32 (adler32-zlib) [ 420.098350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 420.108193] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 422.909150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.113@tcp [ 423.123844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.113@tcp -m GET -e local_interrupt -r 1 [ 423.138957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.13@tcp -m GET -e local_interrupt -r 1 [ 423.153222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.113@tcp [ 423.160749] LNet: There was an unexpected network error while writing to 192.168.201.113: rc = -22 [ 423.199275] LNet: 1 local NIs in recovery (showing 1): 192.168.201.13@tcp [ 423.238879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 423.250887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 423.264529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 423.278319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 423.290186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 423.301392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 423.312349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 423.325161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 423.332584] LNet: 57994:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 423.340228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 423.352668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 423.366309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 423.381727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 423.396439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 423.411109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 423.425378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 423.614864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.113@tcp -m GET -e local_dropped -r 1 [ 423.630887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp -d 192.168.201.13@tcp -m GET -e local_dropped -r 1 [ 423.645032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.113@tcp [ 423.736704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 423.749403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 423.761428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 423.775263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 423.787788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 423.800326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 423.813420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 423.831671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 423.840647] LNet: 58103:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 423.849266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 423.863363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 423.876905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 423.892199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 423.907130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 423.921508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 423.935721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 424.130281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp -d 192.168.201.113@tcp -m GET -e local_aborted -r 1 [ 424.146285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp -d 192.168.201.13@tcp -m GET -e local_aborted -r 1 [ 424.164573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.113@tcp [ 424.175592] LNet: There was an unexpected network error while writing to 192.168.201.113: rc = -22 [ 424.178700] LNet: Skipped 1 previous similar message [ 424.256604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 424.269118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 424.283296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 424.297151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 424.309924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 424.322766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 424.336277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 424.349251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 424.363286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 424.378157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 424.395110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 424.409353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 424.424243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 424.439672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 424.454939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 424.631995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.113@tcp -m GET -e local_no_route -r 1 [ 424.646800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp -d 192.168.201.13@tcp -m GET -e local_no_route -r 1 [ 424.659710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.113@tcp [ 424.739565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 424.752431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 424.767427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 424.781902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 424.795557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 424.809242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 424.823193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 424.836855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 424.843750] LNet: 58321:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 424.846250] LNet: 58321:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 424.853377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 424.865526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 424.882234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 424.896229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 424.908087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 424.920742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 424.934906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 425.112354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp -d 192.168.201.113@tcp -m GET -e local_timeout -r 1 [ 425.128277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.13@tcp -m GET -e local_timeout -r 1 [ 425.145574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.113@tcp [ 425.228421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 425.242110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 425.255113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 425.267808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 425.279897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 425.293679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 425.307492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 425.319879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 425.333389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 425.346987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 425.359918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 425.372838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 425.385779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 425.398995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 425.414360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 425.602810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp -d 192.168.201.113@tcp -m GET -e local_error -r 1 [ 425.619595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp -d 192.168.201.13@tcp -m GET -e local_error -r 1 [ 425.637856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.113@tcp [ 425.647918] LNet: There was an unexpected network error while writing to 192.168.201.113: rc = -22 [ 425.651235] LNet: Skipped 3 previous similar messages [ 425.723509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 425.736466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 425.749505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 425.765751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 425.781388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 425.796209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 425.813225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 425.828561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 425.844117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 425.856321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 425.868469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 425.881094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 425.894221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 425.907268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 425.922045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 426.015473] LNet: 58580:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 426.018434] LNetError: 58580:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 426.029293] LNet: Removed LNI 192.168.201.13@tcp [ 426.192467] Key type .llcrypt unregistered [ 426.193745] Key type ._llcrypt unregistered [ 431.410177] Key type ._llcrypt registered [ 431.411317] Key type .llcrypt registered [ 431.918979] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 07:44:50 (1784288690) [ 432.199424] Key type .llcrypt unregistered [ 432.200874] Key type ._llcrypt unregistered [ 434.185529] Key type ._llcrypt registered [ 434.186634] Key type .llcrypt registered [ 434.249022] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 434.254537] alg: No test for adler32 (adler32-zlib) [ 435.059256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 435.071381] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 437.992525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.113@tcp [ 441.193717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 441.208310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 441.217551] LNet: Added LNI 192.168.201.13@tcp1 [8/256/0/180] [ 441.673768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.113@tcp -m GET -e local_interrupt -r 1 [ 441.689165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp -d 192.168.201.113@tcp1 -m GET -e local_interrupt -r 1 [ 441.703482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp -d 192.168.201.13@tcp -m GET -e local_interrupt -r 1 [ 441.719470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp -d 192.168.201.13@tcp1 -m GET -e local_interrupt -r 1 [ 441.735083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp1 -d 192.168.201.113@tcp -m GET -e local_interrupt -r 1 [ 441.751736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp1 -d 192.168.201.113@tcp1 -m GET -e local_interrupt -r 1 [ 441.768684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp1 -d 192.168.201.13@tcp -m GET -e local_interrupt -r 1 [ 441.788412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp1 -d 192.168.201.13@tcp1 -m GET -e local_interrupt -r 1 [ 441.805198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.113@tcp [ 441.813338] LNet: There was an unexpected network error while writing to 192.168.201.113: rc = -22 [ 441.816417] LNet: 1 local NIs in recovery (showing 1): 192.168.201.13@tcp [ 441.893807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 441.907460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 441.921527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 441.937735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 441.952163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 441.964916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 441.977557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 441.991916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 442.000654] LNet: 60041:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 442.010887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 442.024691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 442.038705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 442.051498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 442.065908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 442.080446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 442.097798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 442.284502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp -d 192.168.201.113@tcp -m GET -e local_dropped -r 1 [ 442.300164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp -d 192.168.201.113@tcp1 -m GET -e local_dropped -r 1 [ 442.316900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp -d 192.168.201.13@tcp -m GET -e local_dropped -r 1 [ 442.334163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.13@tcp1 -m GET -e local_dropped -r 1 [ 442.351660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp1 -d 192.168.201.113@tcp -m GET -e local_dropped -r 1 [ 442.369132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp1 -d 192.168.201.113@tcp1 -m GET -e local_dropped -r 1 [ 442.394462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp1 -d 192.168.201.13@tcp -m GET -e local_dropped -r 1 [ 442.413356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp1 -d 192.168.201.13@tcp1 -m GET -e local_dropped -r 1 [ 442.431586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.113@tcp [ 442.440559] LNet: There was an unexpected network error while writing to 192.168.201.113: rc = -22 [ 442.443161] LNet: Skipped 2 previous similar messages [ 442.520121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 442.533812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 442.547748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 442.562547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 442.575631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 442.589839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 442.603481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 442.615717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 442.622388] LNet: 60162:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 442.629951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 442.644222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 442.657441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 442.672376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 442.686883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 442.700048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 442.713993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 442.899513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.113@tcp -m GET -e local_aborted -r 1 [ 442.920618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.113@tcp1 -m GET -e local_aborted -r 1 [ 442.938813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp -d 192.168.201.13@tcp -m GET -e local_aborted -r 1 [ 442.955977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp -d 192.168.201.13@tcp1 -m GET -e local_aborted -r 1 [ 442.971966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp1 -d 192.168.201.113@tcp -m GET -e local_aborted -r 1 [ 442.987912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp1 -d 192.168.201.113@tcp1 -m GET -e local_aborted -r 1 [ 443.005972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp1 -d 192.168.201.13@tcp -m GET -e local_aborted -r 1 [ 443.022739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp1 -d 192.168.201.13@tcp1 -m GET -e local_aborted -r 1 [ 443.040170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.113@tcp [ 443.131356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 443.143260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 443.155635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 443.166890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 443.179606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 443.189899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 443.202496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 443.215648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 443.228711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 443.240204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 443.253128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 443.266288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 443.279753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 443.299053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 443.314346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 443.496261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.113@tcp -m GET -e local_no_route -r 1 [ 443.510284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.113@tcp1 -m GET -e local_no_route -r 1 [ 443.525995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.13@tcp -m GET -e local_no_route -r 1 [ 443.547242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp -d 192.168.201.13@tcp1 -m GET -e local_no_route -r 1 [ 443.564254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp1 -d 192.168.201.113@tcp -m GET -e local_no_route -r 1 [ 443.580699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp1 -d 192.168.201.113@tcp1 -m GET -e local_no_route -r 1 [ 443.596242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp1 -d 192.168.201.13@tcp -m GET -e local_no_route -r 1 [ 443.610617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp1 -d 192.168.201.13@tcp1 -m GET -e local_no_route -r 1 [ 443.623610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.113@tcp [ 443.632855] LNet: There was an unexpected network error while writing to 192.168.201.113: rc = -22 [ 443.636030] LNet: Skipped 7 previous similar messages [ 443.712659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 443.726143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 443.741143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 443.755184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 443.769680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 443.782734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 443.797768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 443.810681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 443.817590] LNet: 60405:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 443.820851] LNet: 60405:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 443.828808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 443.841554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 443.856355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 443.868728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 443.883183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 443.898687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 443.911312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 444.086402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.113@tcp -m GET -e local_timeout -r 1 [ 444.101827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp -d 192.168.201.113@tcp1 -m GET -e local_timeout -r 1 [ 444.116340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp -d 192.168.201.13@tcp -m GET -e local_timeout -r 1 [ 444.132095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp -d 192.168.201.13@tcp1 -m GET -e local_timeout -r 1 [ 444.147963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp1 -d 192.168.201.113@tcp -m GET -e local_timeout -r 1 [ 444.161906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp1 -d 192.168.201.113@tcp1 -m GET -e local_timeout -r 1 [ 444.176536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp1 -d 192.168.201.13@tcp -m GET -e local_timeout -r 1 [ 444.190918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp1 -d 192.168.201.13@tcp1 -m GET -e local_timeout -r 1 [ 444.204448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.113@tcp [ 444.284206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 444.295828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 444.307592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 444.320650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 444.334108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 444.348109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 444.360826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 444.373565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 444.387480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 444.401495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 444.414304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 444.426772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 444.439718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 444.452060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 444.465914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 444.641458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp -d 192.168.201.113@tcp -m GET -e local_error -r 1 [ 444.656504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.113@tcp1 -m GET -e local_error -r 1 [ 444.669885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.13@tcp -m GET -e local_error -r 1 [ 444.685094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp -d 192.168.201.13@tcp1 -m GET -e local_error -r 1 [ 444.698595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp1 -d 192.168.201.113@tcp -m GET -e local_error -r 1 [ 444.713506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp1 -d 192.168.201.113@tcp1 -m GET -e local_error -r 1 [ 444.727629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp1 -d 192.168.201.13@tcp -m GET -e local_error -r 1 [ 444.741605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp1 -d 192.168.201.13@tcp1 -m GET -e local_error -r 1 [ 444.756282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.113@tcp [ 444.831619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 444.843950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 444.857940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 444.870151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 444.883305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 444.896091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 444.907946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 444.920210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 444.933453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 444.945559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 444.959049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 444.972950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 444.987282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 445.003399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 445.017375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 447.157381] LNet: 60723:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 447.160399] LNetError: 60723:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 447.163737] LNet: Removed LNI 192.168.201.13@tcp [ 448.229325] LNet: Removed LNI 192.168.201.13@tcp1 [ 448.364559] Key type .llcrypt unregistered [ 448.365619] Key type ._llcrypt unregistered [ 453.583817] Key type ._llcrypt registered [ 453.585186] Key type .llcrypt registered [ 454.042493] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 07:45:12 (1784288712) [ 454.311256] Key type .llcrypt unregistered [ 454.312458] Key type ._llcrypt unregistered [ 456.248366] Key type ._llcrypt registered [ 456.249691] Key type .llcrypt registered [ 456.325189] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 456.331119] alg: No test for adler32 (adler32-zlib) [ 457.143585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 457.154719] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 460.149054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.113@tcp [ 460.392920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp -d 192.168.201.113@tcp -m GET -e remote_dropped -r 1 [ 460.410335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.13@tcp -m GET -e remote_dropped -r 1 [ 460.426631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.113@tcp [ 460.435729] LNet: There was an unexpected network error while writing to 192.168.201.113: rc = -22 [ 460.514827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 460.528300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 460.541435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 460.554820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 460.566874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 460.579123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 460.591741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 460.606760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 460.615128] LNet: 61968:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 460.623462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 460.636659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 460.650663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 460.666633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 460.685611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 460.700922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 460.715603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 460.909260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.113@tcp -m GET -e remote_error -r 1 [ 460.924511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp -d 192.168.201.13@tcp -m GET -e remote_error -r 1 [ 460.941400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.113@tcp [ 460.951107] LNet: There was an unexpected network error while writing to 192.168.201.113: rc = -22 [ 461.042865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 461.055605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 461.070973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 461.084971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 461.098370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 461.114685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 461.127542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 461.139574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 461.147632] LNet: 62077:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 461.156780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 461.171222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 461.185058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 461.199931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 461.212783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 461.225877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 461.236587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 461.417772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.113@tcp -m GET -e remote_timeout -r 1 [ 461.438585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.13@tcp -m GET -e remote_timeout -r 1 [ 461.456208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.113@tcp [ 461.539282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 461.555138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 461.568279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 461.581717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 461.595548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 461.607296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 461.619658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 461.634393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 461.647330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 461.662305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 461.675626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 461.690801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 461.707909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 461.723614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 461.739193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 461.837584] LNet: 62227:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 461.840444] LNetError: 62227:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 461.849591] LNet: Removed LNI 192.168.201.13@tcp [ 461.985468] Key type .llcrypt unregistered [ 461.986773] Key type ._llcrypt unregistered [ 467.276750] Key type ._llcrypt registered [ 467.278228] Key type .llcrypt registered [ 467.709722] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 07:45:26 (1784288726) [ 467.985214] Key type .llcrypt unregistered [ 467.986268] Key type ._llcrypt unregistered [ 469.834689] Key type ._llcrypt registered [ 469.835810] Key type .llcrypt registered [ 469.904213] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 469.910066] alg: No test for adler32 (adler32-zlib) [ 470.713784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 470.724438] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 473.453752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.113@tcp [ 475.946229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 475.961460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 475.969026] LNet: Added LNI 192.168.201.13@tcp1 [8/256/0/180] [ 476.400918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp -d 192.168.201.113@tcp -m GET -e remote_dropped -r 1 [ 476.415424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.113@tcp1 -m GET -e remote_dropped -r 1 [ 476.430447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.13@tcp -m GET -e remote_dropped -r 1 [ 476.444653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp -d 192.168.201.13@tcp1 -m GET -e remote_dropped -r 1 [ 476.457632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp1 -d 192.168.201.113@tcp -m GET -e remote_dropped -r 1 [ 476.471438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp1 -d 192.168.201.113@tcp1 -m GET -e remote_dropped -r 1 [ 476.487112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp1 -d 192.168.201.13@tcp -m GET -e remote_dropped -r 1 [ 476.500673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp1 -d 192.168.201.13@tcp1 -m GET -e remote_dropped -r 1 [ 476.513712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.113@tcp [ 476.522584] LNet: There was an unexpected network error while writing to 192.168.201.113: rc = -22 [ 476.525387] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.113@tcp [ 476.588365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 476.599097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 476.615356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 476.627742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 476.638346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 476.650097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 476.660469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 476.670588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 476.676595] LNet: 63688:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 476.682822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 476.693252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 476.703491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 476.715968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 476.728480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 476.738583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 476.749253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 476.760808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 476.922631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp -d 192.168.201.113@tcp -m GET -e remote_error -r 1 [ 476.935507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.113@tcp1 -m GET -e remote_error -r 1 [ 476.948797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp -d 192.168.201.13@tcp -m GET -e remote_error -r 1 [ 476.960473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.13@tcp1 -m GET -e remote_error -r 1 [ 476.974252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp1 -d 192.168.201.113@tcp -m GET -e remote_error -r 1 [ 476.987589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp1 -d 192.168.201.113@tcp1 -m GET -e remote_error -r 1 [ 477.002057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp1 -d 192.168.201.13@tcp -m GET -e remote_error -r 1 [ 477.016128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp1 -d 192.168.201.13@tcp1 -m GET -e remote_error -r 1 [ 477.030134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.113@tcp [ 477.037114] LNet: There was an unexpected network error while writing to 192.168.201.113: rc = -22 [ 477.039268] LNet: Skipped 2 previous similar messages [ 477.106136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 477.116507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 477.127371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 477.139391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 477.153381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 477.167836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 477.181418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 477.192699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 477.200650] LNet: 63811:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 477.208143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 477.221537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 477.233232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 477.244265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 477.254804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 477.265863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 477.277496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 477.288905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 477.447506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp -d 192.168.201.113@tcp -m GET -e remote_timeout -r 1 [ 477.462657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.113@tcp1 -m GET -e remote_timeout -r 1 [ 477.477338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.13@tcp -m GET -e remote_timeout -r 1 [ 477.491085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.13@tcp1 -m GET -e remote_timeout -r 1 [ 477.504893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp1 -d 192.168.201.113@tcp -m GET -e remote_timeout -r 1 [ 477.517987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp1 -d 192.168.201.113@tcp1 -m GET -e remote_timeout -r 1 [ 477.531874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp1 -d 192.168.201.13@tcp -m GET -e remote_timeout -r 1 [ 477.545688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp1 -d 192.168.201.13@tcp1 -m GET -e remote_timeout -r 1 [ 477.561430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.113@tcp [ 477.644341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 477.657982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 477.669702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 477.680547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 477.691721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 477.703637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 477.715967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 477.729741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 477.743964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 477.755514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 477.766282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 477.776760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 477.790521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 477.802659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 477.813704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 477.825681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 479.999520] LNet: 64012:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 480.002355] LNetError: 64012:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 480.006168] LNet: Removed LNI 192.168.201.13@tcp [ 480.019181] LNet: Removed LNI 192.168.201.13@tcp1 [ 480.149550] Key type .llcrypt unregistered [ 480.151051] Key type ._llcrypt unregistered [ 485.143073] Key type ._llcrypt registered [ 485.144409] Key type .llcrypt registered [ 485.563766] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 07:45:44 (1784288744) [ 486.053971] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 486.618119] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 07:45:45 (1784288745) [ 486.854294] Key type .llcrypt unregistered [ 486.855829] Key type ._llcrypt unregistered [ 488.691017] Key type ._llcrypt registered [ 488.692577] Key type .llcrypt registered [ 488.750163] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 488.756868] alg: No test for adler32 (adler32-zlib) [ 489.557114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 489.567609] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 492.445279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.113@tcp [ 492.673798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp -d 192.168.201.113@tcp -m GET -e network_timeout -r 1 [ 492.690571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp -d 192.168.201.13@tcp -m GET -e network_timeout -r 1 [ 492.707683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.113@tcp [ 492.717029] LNet: There was an unexpected network error while writing to 192.168.201.113: rc = -22 [ 492.799812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 492.812480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 492.825367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 492.836530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 492.849469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 492.863361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 492.876907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 492.889405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 492.896859] LNet: 65551:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 492.904257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 492.917764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 492.930766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 492.944314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 492.960544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 492.977275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 493.077501] LNet: 65590:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 493.080653] LNetError: 65590:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 493.088417] LNet: Removed LNI 192.168.201.13@tcp [ 493.222523] Key type .llcrypt unregistered [ 493.223459] Key type ._llcrypt unregistered [ 498.702799] Key type ._llcrypt registered [ 498.704123] Key type .llcrypt registered [ 498.764977] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 498.769826] alg: No test for adler32 (adler32-zlib) [ 499.574219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 499.584826] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 502.356065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.113@tcp [ 504.906312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 504.919992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 504.927693] LNet: Added LNI 192.168.201.13@tcp1 [8/256/0/180] [ 505.402178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.113@tcp -m GET -e network_timeout -r 1 [ 505.419939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp -d 192.168.201.113@tcp1 -m GET -e network_timeout -r 1 [ 505.435073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp -d 192.168.201.13@tcp -m GET -e network_timeout -r 1 [ 505.449138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.13@tcp1 -m GET -e network_timeout -r 1 [ 505.464253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp1 -d 192.168.201.113@tcp -m GET -e network_timeout -r 1 [ 505.481137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp1 -d 192.168.201.113@tcp1 -m GET -e network_timeout -r 1 [ 505.497440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp1 -d 192.168.201.13@tcp -m GET -e network_timeout -r 1 [ 505.511670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp1 -d 192.168.201.13@tcp1 -m GET -e network_timeout -r 1 [ 505.525908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.113@tcp [ 505.534406] LNet: There was an unexpected network error while writing to 192.168.201.113: rc = -22 [ 505.607446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 505.618915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 505.630920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 505.644701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 505.657694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 505.672787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 505.688981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 505.706252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 505.714823] LNet: 66465:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 505.723608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 505.737436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 505.749474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 505.759104] LNet: 1 local NIs in recovery (showing 1): 192.168.201.13@tcp [ 505.761560] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.113@tcp [ 505.762370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 505.777451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 505.793153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 507.865876] LNet: There was an unexpected network error while writing to 192.168.201.113: rc = -22 [ 507.960715] LNet: 66538:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 507.963516] LNetError: 66538:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 507.966850] LNet: Removed LNI 192.168.201.13@tcp [ 507.981291] LNet: Removed LNI 192.168.201.13@tcp1 [ 508.116733] Key type .llcrypt unregistered [ 508.117869] Key type ._llcrypt unregistered [ 513.241801] Key type ._llcrypt registered [ 513.242639] Key type .llcrypt registered [ 513.698301] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 07:46:12 (1784288772) [ 513.809887] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 513.814856] alg: No test for adler32 (adler32-zlib) [ 514.619734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 514.641957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 514.653289] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 514.906445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 514.915927] LNet: Added LNI 192.168.201.13@tcp1 [8/256/0/180] [ 515.236294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.13@tcp [ 515.263601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 515.281612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 515.296578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 515.312187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 515.323984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 515.337513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.13@tcp [ 515.345374] LNetError: 67635:0:(api-ni.c:4418:lnet_ni_set_healthv()) manually adding local NI 192.168.201.13@tcp to recovery [ 515.679327] LNet: 1 local NIs in recovery (showing 1): 192.168.201.13@tcp [ 515.679361] LNet: There was an unexpected network error while writing to 192.168.201.13: rc = -22 [ 517.727390] LNet: There was an unexpected network error while writing to 192.168.201.13: rc = -22 [ 518.498387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 521.823314] LNet: There was an unexpected network error while writing to 192.168.201.13: rc = -22 [ 522.690214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 522.705577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 522.730204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 522.738460] LNet: 67752:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 522.741048] LNetError: 67752:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 522.743970] LNet: Removed LNI 192.168.201.13@tcp [ 522.756412] LNet: Removed LNI 192.168.201.13@tcp1 [ 522.767179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 522.790533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 522.801368] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 523.053545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 523.061520] LNet: Added LNI 192.168.201.13@tcp1 [8/256/0/180] [ 523.357115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.13@tcp [ 523.370666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 523.391305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 523.408615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 523.422940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 523.437228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 523.450408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 523.463487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.13@tcp [ 523.471225] LNetError: 68024:0:(api-ni.c:4418:lnet_ni_set_healthv()) manually adding local NI 192.168.201.13@tcp to recovery [ 525.855278] LNet: There was an unexpected network error while writing to 192.168.201.13: rc = -22 [ 525.857474] LNet: Skipped 1 previous similar message [ 526.623423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 529.951315] LNet: There was an unexpected network error while writing to 192.168.201.13: rc = -22 [ 533.897216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 533.911395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 533.924869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 536.139984] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 07:46:34 (1784288794) [ 536.175900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 536.185234] LNet: 68736:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 536.188703] LNetError: 68736:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 536.191183] LNetError: 68736:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 536.194953] LNet: Removed LNI 192.168.201.13@tcp [ 536.202850] LNet: Removed LNI 192.168.201.13@tcp1 [ 536.210637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 536.230388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 536.240099] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 536.488704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 536.497136] LNet: Added LNI 192.168.201.13@tcp1 [8/256/0/180] [ 536.770534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.13@tcp [ 536.800864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 536.814439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 536.829675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 536.841650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 536.853993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 536.865893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.13@tcp --health 0 [ 536.902952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 537.247243] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.13@tcp [ 539.295291] LNet: There was an unexpected network error while writing to 192.168.201.13: rc = -22 [ 539.298108] LNet: Skipped 2 previous similar messages [ 552.317535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 552.371129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 552.384773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 552.396742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.13@tcp --health 500 [ 552.430280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 559.667440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 559.696802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 559.707752] LNet: 69247:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 559.711176] LNetError: 69247:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 559.713500] LNetError: 69247:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 559.716381] LNet: Removed LNI 192.168.201.13@tcp [ 559.728370] LNet: Removed LNI 192.168.201.13@tcp1 [ 559.737600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 559.766585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 559.776466] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 560.045988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 560.056738] LNet: Added LNI 192.168.201.13@tcp1 [8/256/0/180] [ 560.357981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.13@tcp [ 560.372282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 560.396765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 560.411322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 560.424700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 560.439444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 560.455615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 560.468708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.13@tcp --health 0 [ 561.823346] LNet: There was an unexpected network error while writing to 192.168.201.13: rc = -22 [ 561.825888] LNet: Skipped 2 previous similar messages [ 562.595518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 571.950086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 571.965077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 571.978808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 574.213398] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 07:47:13 (1784288833) [ 574.360056] LNet: 70302:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 574.362746] LNetError: 70302:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 574.365812] LNetError: 70302:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 574.370054] LNet: Removed LNI 192.168.201.13@tcp [ 574.378570] LNet: Removed LNI 192.168.201.13@tcp1 [ 574.621922] Key type .llcrypt unregistered [ 574.623202] Key type ._llcrypt unregistered [ 576.739105] Key type ._llcrypt registered [ 576.740414] Key type .llcrypt registered [ 576.806304] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 576.812612] alg: No test for adler32 (adler32-zlib) [ 577.618636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 577.631904] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 580.889809] Lustre: DEBUG MARKER: Initial discovery [ 580.912399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.113@tcp [ 581.669882] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 581.686640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 581.704563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.113@tcp [ 581.714342] LNet: There was an unexpected network error while writing to 192.168.201.113: rc = -22 [ 581.723594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 581.727076] LNet: 1 local NIs in recovery (showing 1): 192.168.201.13@tcp [ 581.738598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.201.113@tcp -r 1 -m GET -l 3 [ 581.752595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.201.113@tcp -r 1 -m GET -e local_error [ 581.766733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.201.113@tcp -r 1 -m PUT -l 6 [ 582.252364] Lustre: DEBUG MARKER: Force oleg113-client.virtnet to discover 192.168.201.113@tcp (in background) [ 582.272492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.113@tcp [ 582.734099] Lustre: DEBUG MARKER: Force oleg113-server to discover 192.168.201.13@tcp [ 583.508942] Lustre: DEBUG MARKER: Wait for 71055 [ 585.311350] LNet: There was an unexpected network error while writing to 192.168.201.113: rc = -22 [ 585.773895] Lustre: DEBUG MARKER: Finished wait on 71055 [ 585.798060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 585.813299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 585.903293] LNet: 71512:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 585.905586] LNetError: 71512:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 585.913345] LNet: Removed LNI 192.168.201.13@tcp [ 586.048373] Key type .llcrypt unregistered [ 586.049558] Key type ._llcrypt unregistered [ 591.367827] Key type ._llcrypt registered [ 591.369304] Key type .llcrypt registered [ 591.817987] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 07:47:30 (1784288850) [ 591.980230] Key type .llcrypt unregistered [ 591.981380] Key type ._llcrypt unregistered [ 593.809903] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 593.850189] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 593.853057] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 593.855520] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 596.064506] Key type ._llcrypt registered [ 596.065584] Key type .llcrypt registered [ 596.139284] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 596.147111] alg: No test for adler32 (adler32-zlib) [ 596.954501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 596.976995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 596.986359] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 597.244789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 597.254457] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 600.008839] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 07:47:38 (1784288858) [ 600.041948] LNet: 73579:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 600.044995] LNetError: 73579:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 600.048169] LNet: Removed LNI 10.1.2.3@tcp [ 600.053807] LNet: Removed LNI 192.168.201.13@tcp [ 600.298695] Key type .llcrypt unregistered [ 600.300280] Key type ._llcrypt unregistered [ 602.287875] Key type ._llcrypt registered [ 602.289091] Key type .llcrypt registered [ 602.348885] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 602.355403] alg: No test for adler32 (adler32-zlib) [ 603.152848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 603.171914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 603.180180] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 603.408168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 603.416605] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 603.745672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.201.13@tcp [ 604.767576] LNetError: Unexpected error -22 connecting to 192.168.201.13@tcp at host 192.168.201.13:988 [ 605.920191] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 07:47:44 (1784288864) [ 605.952280] LNet: 74707:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 605.954296] LNetError: 74707:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 605.957604] LNet: Removed LNI 10.1.2.3@tcp [ 605.965712] LNet: Removed LNI 192.168.201.13@tcp [ 606.205672] Key type .llcrypt unregistered [ 606.206755] Key type ._llcrypt unregistered [ 608.225435] Key type ._llcrypt registered [ 608.226503] Key type .llcrypt registered [ 608.293127] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 608.299850] alg: No test for adler32 (adler32-zlib) [ 609.100771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 609.122867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 609.132565] LNet: Added LNI 192.168.201.13@tcp1 [8/256/0/180] [ 609.418283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 609.426247] LNet: Added LNI 192.168.201.13@tcp2 [8/256/0/180] [ 609.750771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.201.13@tcp1 --nid 192.168.201.13@tcp2 [ 611.213432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.13@tcp1 192.168.201.13@tcp2 [ 611.221894] LNetError: 75548:0:(lib-move.c:2063:lnet_handle_find_routed_path()) Unable to send message from 192.168.201.13@tcp1 to 192.168.201.13@tcp2 - Route table may be misconfigured [ 611.233550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.13@tcp2 192.168.201.13@tcp1 [ 613.309912] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 07:47:52 (1784288872) [ 613.338653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 613.348196] LNet: 76115:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 613.350968] LNetError: 76115:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 613.353506] LNet: Removed LNI 192.168.201.13@tcp1 [ 614.373569] LNet: Removed LNI 192.168.201.13@tcp2 [ 614.381905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 614.402347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 614.411571] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 614.657301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 614.666309] LNet: Added LNI 192.168.201.13@tcp1 [8/256/0/180] [ 614.958074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.13@tcp [ 614.972716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.13@tcp 192.168.201.13@tcp [ 614.985756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.13@tcp1 192.168.201.13@tcp1 [ 614.999559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.13@tcp -d 192.168.201.13@tcp -e network_timeout [ 615.013561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.13@tcp -d 192.168.201.13@tcp1 -e network_timeout [ 615.029054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.13@tcp1 -d 192.168.201.13@tcp -e network_timeout [ 615.044305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.13@tcp1 -d 192.168.201.13@tcp1 -e network_timeout [ 615.061094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.13@tcp [ 615.069153] LNet: There was an unexpected network error while writing to 192.168.201.13: rc = -22 [ 615.077688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 615.114316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 615.149970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 615.455091] LNet: 1 local NIs in recovery (showing 1): 192.168.201.13@tcp [ 653.899881] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 07:48:32 (1784288912) [ 654.137866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 654.147336] LNet: 76984:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 654.150527] LNetError: 76984:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 654.153927] LNetError: 76984:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 654.156503] LNet: Removed LNI 192.168.201.13@tcp [ 655.205531] LNet: Removed LNI 192.168.201.13@tcp1 [ 655.214837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 655.248505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 655.355238] LNet: 77021:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 655.518700] Key type .llcrypt unregistered [ 655.520269] Key type ._llcrypt unregistered [ 659.323506] Key type ._llcrypt registered [ 659.325057] Key type .llcrypt registered [ 659.753954] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 07:48:38 (1784288918) [ 659.887894] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 659.894503] alg: No test for adler32 (adler32-zlib) [ 660.690737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 661.130645] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 661.599898] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 07:48:40 (1784288920) [ 661.624665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 661.633201] LNet: 78112:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 661.640792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 661.661666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 661.670949] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 661.893813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 661.901463] LNet: Added LNI 192.168.201.13@tcp1 [8/256/0/180] [ 662.202404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.13@tcp [ 662.216722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.13@tcp1 [ 662.230162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.13@tcp1 [ 664.143663] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 07:48:43 (1784288923) [ 664.748929] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg113-server [ 665.285952] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 07:48:44 (1784288924) [ 665.856333] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg113-server [ 666.379114] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 07:48:45 (1784288925) [ 666.976983] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg113-server [ 667.506947] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 07:48:46 (1784288926) [ 668.091872] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg113-server [ 668.624473] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 07:48:47 (1784288927) [ 669.242942] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg113-server [ 669.847992] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 07:48:48 (1784288928) [ 670.479888] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg113-server [ 671.054639] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 07:48:49 (1784288929) [ 671.758310] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg113-server [ 672.402759] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 07:48:51 (1784288931) [ 673.093273] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg113-server [ 673.825944] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 07:48:52 (1784288932) [ 674.513411] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg113-server [ 675.181572] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 07:48:53 (1784288933) [ 675.238939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 675.250129] LNet: 81993:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 675.253506] LNetError: 81993:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 675.257525] LNet: Removed LNI 192.168.201.13@tcp [ 676.325451] LNet: Removed LNI 192.168.201.13@tcp1 [ 676.333870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 676.356763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 676.367459] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 676.638367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 676.668684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.13@tcp [ 676.733396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 676.743136] LNet: 82140:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 676.746262] LNetError: 82140:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 676.749057] LNetError: 82140:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 676.758340] LNet: Removed LNI 192.168.201.13@tcp [ 676.768686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 676.793563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 676.805116] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 677.093744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.13@tcp [ 677.169677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 678.245478] LNet: Removed LNI 192.168.201.13@tcp [ 678.259652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 678.286823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 678.299377] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 678.681587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 678.718790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.13@tcp [ 678.800707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 678.811763] LNet: 82432:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 678.814762] LNet: 82432:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 678.817911] LNetError: 82432:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 678.820022] LNetError: 82432:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 678.831699] LNet: Removed LNI 192.168.201.13@tcp [ 678.840776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 678.868906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 678.881115] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 679.194108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.13@tcp [ 680.351878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 681.385165] LNet: Removed LNI 192.168.201.13@tcp [ 681.395796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 681.419752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 681.430120] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 681.726535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 681.762992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.13@tcp [ 681.846099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 681.856063] LNetError: 82734:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 681.859604] LNetError: 82734:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 681.873048] LNet: Removed LNI 192.168.201.13@tcp [ 681.884790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 681.908111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 681.920674] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 682.239464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.13@tcp [ 682.316840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 683.365462] LNet: Removed LNI 192.168.201.13@tcp [ 683.373856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 683.403139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 683.416969] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 683.727599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 683.765880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.13@tcp [ 683.882371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 683.891685] LNet: 83026:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 683.894303] LNet: 83026:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 684.966648] LNet: Removed LNI 192.168.201.13@tcp [ 684.989844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 685.016612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 685.038543] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 685.424792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.13@tcp [ 686.552273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 686.560493] LNetError: 83181:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 686.562817] LNetError: 83181:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 686.569414] LNet: Removed LNI 192.168.201.13@tcp [ 686.578497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 686.601129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 686.613900] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 686.890573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 686.923763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.13@tcp [ 688.072673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 689.125375] LNet: Removed LNI 192.168.201.13@tcp [ 689.135300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 689.160736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 689.171200] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 689.455195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.13@tcp [ 689.530709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 690.597325] LNet: Removed LNI 192.168.201.13@tcp [ 690.608328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 690.634608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 690.647317] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 690.928750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 690.956167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.13@tcp [ 692.073154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 692.081847] LNet: 83638:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 692.084724] LNet: 83638:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 693.157388] LNet: Removed LNI 192.168.201.13@tcp [ 693.165757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 693.187738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 693.198408] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 693.473476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.13@tcp [ 693.560827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 694.629376] LNet: Removed LNI 192.168.201.13@tcp [ 694.637888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 694.659560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 694.668163] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 694.948716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 694.972934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 694.981581] LNetError: 83916:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 694.985428] LNetError: 83916:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 694.994337] LNet: Removed LNI 192.168.201.13@tcp [ 695.003322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 695.027524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 695.037742] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 695.317656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 695.350599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 695.366354] LNet: Removed LNI 192.168.201.13@tcp [ 695.374271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 695.397766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 695.428450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 695.441593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 695.465124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 695.494834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 695.508612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 695.528609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 695.554594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 695.567556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 695.586232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 695.595834] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 695.851962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 695.868503] LNet: Removed LNI 192.168.201.13@tcp [ 695.876697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 695.901316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 698.223028] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 07:49:17 (1784288957) [ 698.256883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 698.277276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 698.305245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 698.315067] LNet: Added LNI 192.168.201.13@tcp231 [8/256/0/180] [ 698.752510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 698.770386] LNet: Removed LNI 192.168.201.13@tcp231 [ 698.778273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 698.805462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 698.816845] LNet: Added LNI 192.168.201.13@tcp231 [8/256/0/180] [ 698.819739] LNet: 84949:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 698.968231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 698.985694] LNet: Removed LNI 192.168.201.13@tcp231 [ 698.991638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 699.002177] LNet: Added LNI 192.168.201.13@tcp231 [8/256/0/180] [ 699.407449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 699.430323] LNet: Removed LNI 192.168.201.13@tcp231 [ 699.439577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 699.472607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 699.482465] LNet: Added LNI 192.168.201.13@tcp231 [8/256/0/0] [ 699.485258] LNet: 85101:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 699.629536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 699.651305] LNet: Removed LNI 192.168.201.13@tcp231 [ 699.657725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 699.669350] LNet: Added LNI 192.168.201.13@tcp231 [8/256/0/0] [ 700.081205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 700.094271] LNet: Removed LNI 192.168.201.13@tcp231 [ 700.102485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 700.128426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 700.139808] LNet: Added LNI 192.168.201.13@tcp231 [8/256/0/60] [ 700.294810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 700.313492] LNet: Removed LNI 192.168.201.13@tcp231 [ 700.321173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 700.332826] LNet: Added LNI 192.168.201.13@tcp231 [8/256/0/60] [ 700.751565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 700.770442] LNet: Removed LNI 192.168.201.13@tcp231 [ 700.779385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 700.813436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 700.827116] LNet: Added LNI 192.168.201.13@tcp231 [8/256/0/180] [ 700.829540] LNet: 85404:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 700.831884] LNet: 85404:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 701.012618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 701.031833] LNet: Removed LNI 192.168.201.13@tcp231 [ 701.039044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 701.053524] LNet: Added LNI 192.168.201.13@tcp231 [8/256/0/180] [ 703.547482] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 07:49:22 (1784288962) [ 703.575428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 703.589560] LNet: Removed LNI 192.168.201.13@tcp231 [ 703.598834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 703.625781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 703.635820] LNet: Added LNI 192.168.201.13@tcp232 [8/256/0/180] [ 703.911301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 703.929222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 703.957392] LNet: Removed LNI 192.168.201.13@tcp232 [ 703.964824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 703.974980] LNet: Added LNI 192.168.201.13@tcp232 [8/256/0/180] [ 704.240461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 704.257976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 704.276443] LNet: Removed LNI 192.168.201.13@tcp232 [ 704.283169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 704.294144] LNet: Added LNI 192.168.201.13@tcp232 [8/256/0/180] [ 704.586203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 704.602491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 704.615692] LNet: Removed LNI 192.168.201.13@tcp232 [ 704.623392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 704.635886] LNet: Added LNI 192.168.201.13@tcp232 [8/256/0/180] [ 704.907737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 704.925554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 704.943504] LNet: Removed LNI 192.168.201.13@tcp232 [ 704.949922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 707.040291] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 07:49:25 (1784288965) [ 707.393300] Key type .llcrypt unregistered [ 707.394955] Key type ._llcrypt unregistered [ 709.442414] Key type ._llcrypt registered [ 709.443843] Key type .llcrypt registered [ 709.516431] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 709.525945] alg: No test for adler32 (adler32-zlib) [ 710.331611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 710.342181] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 713.219430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.113@tcp [ 716.093055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 716.108524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 716.119237] LNet: Added LNI 192.168.201.13@tcp1 [8/256/0/180] [ 716.629541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.113@tcp -m GET -e local_interrupt -r 2 [ 716.644511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.113@tcp1 -m GET -e local_interrupt -r 2 [ 716.659727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp -d 192.168.201.13@tcp -m GET -e local_interrupt -r 2 [ 716.675772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp -d 192.168.201.13@tcp1 -m GET -e local_interrupt -r 2 [ 716.691792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp1 -d 192.168.201.113@tcp -m GET -e local_interrupt -r 2 [ 716.707792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp1 -d 192.168.201.113@tcp1 -m GET -e local_interrupt -r 2 [ 716.724118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp1 -d 192.168.201.13@tcp -m GET -e local_interrupt -r 2 [ 716.738571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp1 -d 192.168.201.13@tcp1 -m GET -e local_interrupt -r 2 [ 716.753706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.13@tcp 192.168.201.113@tcp [ 716.763832] LNet: There was an unexpected network error while writing to 192.168.201.113: rc = -22 [ 716.766434] LNet: 1 local NIs in recovery (showing 1): 192.168.201.13@tcp [ 716.852351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 716.865321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 716.882360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 716.896633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 716.910118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 716.921584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 716.933819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 716.946471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 716.955479] LNet: 87899:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 716.962891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 716.975324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 716.988232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 717.000896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 717.014908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 717.028906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 717.206554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.113@tcp -m GET -e local_dropped -r 2 [ 717.222290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp -d 192.168.201.113@tcp1 -m GET -e local_dropped -r 2 [ 717.237833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.13@tcp -m GET -e local_dropped -r 2 [ 717.254747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.13@tcp1 -m GET -e local_dropped -r 2 [ 717.273078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp1 -d 192.168.201.113@tcp -m GET -e local_dropped -r 2 [ 717.289078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp1 -d 192.168.201.113@tcp1 -m GET -e local_dropped -r 2 [ 717.304488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp1 -d 192.168.201.13@tcp -m GET -e local_dropped -r 2 [ 717.320494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp1 -d 192.168.201.13@tcp1 -m GET -e local_dropped -r 2 [ 717.335551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.13@tcp 192.168.201.113@tcp [ 717.344246] LNet: There was an unexpected network error while writing to 192.168.201.113: rc = -22 [ 717.435784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 717.447644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 717.459788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 717.473579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 717.486229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 717.497514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 717.509193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 717.521553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 717.529306] LNet: 88019:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 717.537226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 717.551854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 717.566416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 717.580476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 717.595407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 717.608936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 717.782550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp -d 192.168.201.113@tcp -m GET -e local_aborted -r 2 [ 717.798572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp -d 192.168.201.113@tcp1 -m GET -e local_aborted -r 2 [ 717.812084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.13@tcp -m GET -e local_aborted -r 2 [ 717.827690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp -d 192.168.201.13@tcp1 -m GET -e local_aborted -r 2 [ 717.842360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp1 -d 192.168.201.113@tcp -m GET -e local_aborted -r 2 [ 717.858667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp1 -d 192.168.201.113@tcp1 -m GET -e local_aborted -r 2 [ 717.874961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp1 -d 192.168.201.13@tcp -m GET -e local_aborted -r 2 [ 717.890650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp1 -d 192.168.201.13@tcp1 -m GET -e local_aborted -r 2 [ 717.906580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.13@tcp 192.168.201.113@tcp [ 717.994492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 718.009295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 718.022099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 718.033361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 718.046571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 718.058480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 718.071995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 718.084893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 718.098247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 718.110328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 718.122902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 718.137874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 718.153330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 718.167974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 718.349668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp -d 192.168.201.113@tcp -m GET -e local_no_route -r 2 [ 718.365866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.113@tcp1 -m GET -e local_no_route -r 2 [ 718.381903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp -d 192.168.201.13@tcp -m GET -e local_no_route -r 2 [ 718.397522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.13@tcp1 -m GET -e local_no_route -r 2 [ 718.413713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp1 -d 192.168.201.113@tcp -m GET -e local_no_route -r 2 [ 718.430991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp1 -d 192.168.201.113@tcp1 -m GET -e local_no_route -r 2 [ 718.447882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp1 -d 192.168.201.13@tcp -m GET -e local_no_route -r 2 [ 718.464216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp1 -d 192.168.201.13@tcp1 -m GET -e local_no_route -r 2 [ 718.481628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.13@tcp 192.168.201.113@tcp [ 718.492194] LNet: There was an unexpected network error while writing to 192.168.201.113: rc = -22 [ 718.494784] LNet: Skipped 2 previous similar messages [ 718.573628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 718.584783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 718.598661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 718.612742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 718.627733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 718.641318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 718.655226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 718.668331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 718.677129] LNet: 88259:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 718.680173] LNet: 88259:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 718.689161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 718.701886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 718.714179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 718.726498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 718.740400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 718.755076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 718.934931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp -d 192.168.201.113@tcp -m GET -e local_timeout -r 2 [ 718.950858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp -d 192.168.201.113@tcp1 -m GET -e local_timeout -r 2 [ 718.965790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.13@tcp -m GET -e local_timeout -r 2 [ 718.982889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.13@tcp1 -m GET -e local_timeout -r 2 [ 718.999887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp1 -d 192.168.201.113@tcp -m GET -e local_timeout -r 2 [ 719.015812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp1 -d 192.168.201.113@tcp1 -m GET -e local_timeout -r 2 [ 719.032598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp1 -d 192.168.201.13@tcp -m GET -e local_timeout -r 2 [ 719.048951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp1 -d 192.168.201.13@tcp1 -m GET -e local_timeout -r 2 [ 719.067210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.13@tcp 192.168.201.113@tcp [ 719.155570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 719.168072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 719.180529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 719.193949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 719.206057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 719.219373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 719.232367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 719.245441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 719.259075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 719.272245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 719.286032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 719.299096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 719.312663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 719.325491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 719.512313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.113@tcp -m GET -e remote_dropped -r 2 [ 719.527795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp -d 192.168.201.113@tcp1 -m GET -e remote_dropped -r 2 [ 719.541142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.13@tcp -m GET -e remote_dropped -r 2 [ 719.556680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.13@tcp1 -m GET -e remote_dropped -r 2 [ 719.571090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp1 -d 192.168.201.113@tcp -m GET -e remote_dropped -r 2 [ 719.586269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp1 -d 192.168.201.113@tcp1 -m GET -e remote_dropped -r 2 [ 719.602968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp1 -d 192.168.201.13@tcp -m GET -e remote_dropped -r 2 [ 719.619988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp1 -d 192.168.201.13@tcp1 -m GET -e remote_dropped -r 2 [ 719.635698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.13@tcp 192.168.201.113@tcp [ 719.645435] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.113@tcp [ 719.719540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 719.730965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 719.742697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 719.755294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 719.767351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 719.779422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 719.791896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 719.805176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 719.817109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 719.828242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 719.839463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 719.851477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 719.863953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 719.878442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 722.031579] LNet: 88573:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 722.034186] LNetError: 88573:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 722.037748] LNet: Removed LNI 192.168.201.13@tcp [ 722.047326] LNet: Removed LNI 192.168.201.13@tcp1 [ 722.187524] Key type .llcrypt unregistered [ 722.188746] Key type ._llcrypt unregistered [ 727.178144] Key type ._llcrypt registered [ 727.179402] Key type .llcrypt registered [ 727.611980] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 07:49:46 (1784288986) [ 727.719657] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 727.724657] alg: No test for adler32 (adler32-zlib) [ 728.525543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 728.548586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 728.559471] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 728.840974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 728.853897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.13@tcp [ 728.864037] LNetError: 89549:0:(api-ni.c:4418:lnet_ni_set_healthv()) manually adding local NI 192.168.201.13@tcp to recovery [ 728.897211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.201.13@tcp [ 728.936059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 731.079844] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 07:49:49 (1784288989) [ 731.112595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 731.124324] LNet: 90131:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 731.128039] LNetError: 90131:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 731.139652] LNet: Removed LNI 192.168.201.13@tcp [ 731.150826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 731.180290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 731.190952] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 731.493070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 731.502236] LNet: Added LNI 192.168.201.13@tcp1 [8/256/0/180] [ 731.839676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.201.13@tcp [ 731.848662] LNetError: 90403:0:(api-ni.c:4418:lnet_ni_set_healthv()) manually adding local NI 192.168.201.13@tcp to recovery [ 732.191272] LNet: 1 local NIs in recovery (showing 1): 192.168.201.13@tcp [ 732.191780] LNet: There was an unexpected network error while writing to 192.168.201.13: rc = -22 [ 732.924731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.201.13@tcp [ 736.980471] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 07:49:55 (1784288995) [ 737.574334] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg113-server [ 738.121857] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 07:49:56 (1784288996) [ 738.727883] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg113-server [ 739.327507] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 07:49:58 (1784288998) [ 739.936872] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg113-server [ 740.460469] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 07:49:59 (1784288999) [ 740.490808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 740.500061] LNet: 92025:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 740.502828] LNetError: 92025:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 740.505440] LNet: Removed LNI 192.168.201.13@tcp [ 740.513725] LNet: Removed LNI 192.168.201.13@tcp1 [ 740.524417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 740.546918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 740.556753] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 740.810775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 740.835911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 740.852505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 740.880021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 740.899552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 740.916422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 743.066172] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 07:50:01 (1784289001) [ 743.102981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 743.113907] LNet: 92750:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 743.116214] LNetError: 92750:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 743.119084] LNetError: 92750:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 743.126815] LNet: Removed LNI 192.168.201.13@tcp [ 743.136442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 743.171664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 743.183825] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 745.562331] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 07:50:04 (1784289004) [ 745.592265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 745.603048] LNet: 93452:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 745.606407] LNetError: 93452:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 745.616701] LNet: Removed LNI 192.168.201.13@tcp [ 745.626685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 745.649750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 745.660249] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 748.148722] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 07:50:06 (1784289006) [ 748.649031] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 749.244295] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 07:50:08 (1784289008) [ 749.393045] LNet: Removed LNI 192.168.201.13@tcp [ 749.631629] Key type .llcrypt unregistered [ 749.632993] Key type ._llcrypt unregistered [ 751.643538] Key type ._llcrypt registered [ 751.645143] Key type .llcrypt registered [ 751.715682] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 751.722718] alg: No test for adler32 (adler32-zlib) [ 752.526176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 752.537137] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 755.488846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.113@tcp [ 757.091022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.201.113@tcp [ 757.201894] LNet: 94907:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 757.204341] LNetError: 94907:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 757.214282] LNet: Removed LNI 192.168.201.13@tcp [ 757.368468] Key type .llcrypt unregistered [ 757.369492] Key type ._llcrypt unregistered [ 760.955421] Key type ._llcrypt registered [ 760.956785] Key type .llcrypt registered [ 761.455744] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 762.079638] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 762.693966] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 07:50:21 (1784289021) [ 763.319786] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 763.982452] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 07:50:22 (1784289022) [ 764.628916] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg113-server [ 765.204968] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 07:50:24 (1784289024) [ 765.845842] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg113-server [ 766.401405] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 07:50:25 (1784289025) [ 767.029770] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 767.040823] alg: No test for adler32 (adler32-zlib) [ 767.935641] Lustre: Lustre: Build Version: 2.17.54_173_g0a24909 [ 768.027048] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 769.607164] Key type lgssc registered [ 769.978754] Lustre: Echo OBD driver; http://www.lustre.org/ [ 777.478918] Key type lgssc unregistered [ 777.595502] LNet: 98076:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 777.598874] LNetError: 98076:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 777.610280] LNet: Removed LNI 192.168.201.13@tcp [ 777.851376] Key type .llcrypt unregistered [ 777.852439] Key type ._llcrypt unregistered [ 782.937611] Key type ._llcrypt registered [ 782.938867] Key type .llcrypt registered [ 783.395107] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 07:50:42 (1784289042) [ 783.528336] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 783.534783] alg: No test for adler32 (adler32-zlib) [ 784.337461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 784.365194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 784.384349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 784.404577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 784.423282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 786.634611] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 07:50:45 (1784289045) [ 786.672163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 786.683162] LNet: 99483:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 786.696944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 786.727582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 786.738486] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 787.127574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 787.137961] LNetError: 99632:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 787.145482] LNet: Removed LNI 192.168.201.13@tcp [ 787.154586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 787.177052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 787.187875] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 787.368707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 787.376718] LNet: 99660:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 787.378972] LNet: 99660:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 787.387340] LNet: Removed LNI 192.168.201.13@tcp [ 787.395780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 787.419000] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 787.430088] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 789.688545] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 07:50:48 (1784289048) [ 789.733217] LNet: 100247:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 789.735629] LNetError: 100247:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 789.738053] LNetError: 100247:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 789.746621] LNet: Removed LNI 192.168.201.13@tcp [ 789.970468] Key type .llcrypt unregistered [ 789.972142] Key type ._llcrypt unregistered [ 792.099084] Key type ._llcrypt registered [ 792.100721] Key type .llcrypt registered [ 792.168592] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 792.174781] alg: No test for adler32 (adler32-zlib) [ 792.982738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 793.012082] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 793.013780] LNet: *** cfs_fail_loc=e003, val=100*** [ 793.279397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.13@tcp [ 793.288590] Ping reply received with mapping: [ 793.289772] found metadata reply NI[192.168.201.13@tcp] [ 793.291134] 0d c9 a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 793.292668] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 793.293960] 5a 5a 5a 5a 5a 5a 5a 5a [ 793.345599] LNet: 100647:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 793.348857] LNetError: 100647:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 793.358393] LNet: Removed LNI 192.168.201.13@tcp [ 793.581437] Key type .llcrypt unregistered [ 793.583140] Key type ._llcrypt unregistered [ 797.216651] Key type ._llcrypt registered [ 797.217741] Key type .llcrypt registered [ 797.707671] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 07:50:56 (1784289056) [ 797.870247] Key type .llcrypt unregistered [ 797.871297] Key type ._llcrypt unregistered [ 799.845703] Key type ._llcrypt registered [ 799.847278] Key type .llcrypt registered [ 799.923213] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 799.929353] alg: No test for adler32 (adler32-zlib) [ 800.808797] LNetError: 101690:0:(api-ni.c:2795:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 800.950567] Key type .llcrypt unregistered [ 800.951736] Key type ._llcrypt unregistered [ 802.674225] Key type ._llcrypt registered [ 802.675260] Key type .llcrypt registered [ 803.199969] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 07:51:01 (1784289061) [ 803.391682] Key type .llcrypt unregistered [ 803.393275] Key type ._llcrypt unregistered [ 805.478355] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 805.480547] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 807.652959] Key type ._llcrypt registered [ 807.654234] Key type .llcrypt registered [ 807.726672] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 807.735355] alg: No test for adler32 (adler32-zlib) [ 808.541815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 808.566828] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 812.390101] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 07:51:11 (1784289071) [ 812.421532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 812.430980] LNet: 103463:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 812.434246] LNetError: 103463:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 812.442811] LNet: Removed LNI 10.1.2.3@tcp [ 812.452686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 812.700360] Key type .llcrypt unregistered [ 812.701524] Key type ._llcrypt unregistered [ 814.621048] Key type ._llcrypt registered [ 814.622498] Key type .llcrypt registered [ 814.681806] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 814.687878] alg: No test for adler32 (adler32-zlib) [ 815.496271] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 815.520375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.13@tcp [ 815.576861] LNet: 103755:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 815.579524] LNetError: 103755:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 815.587423] LNet: Removed LNI 192.168.201.13@tcp [ 815.814444] Key type .llcrypt unregistered [ 815.815949] Key type ._llcrypt unregistered [ 819.284944] Key type ._llcrypt registered [ 819.286411] Key type .llcrypt registered [ 819.760369] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 07:51:18 (1784289078) [ 819.962467] Key type .llcrypt unregistered [ 819.963836] Key type ._llcrypt unregistered [ 821.893397] Key type ._llcrypt registered [ 821.895085] Key type .llcrypt registered [ 821.961760] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 821.967240] alg: No test for adler32 (adler32-zlib) [ 824.802338] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 07:51:23 (1784289083) [ 824.847368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 824.862055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 824.885174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 824.894692] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 825.130767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 825.141806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 825.159465] LNet: Removed LNI 192.168.201.13@tcp [ 825.231929] LNet: 105496:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 825.363421] Key type .llcrypt unregistered [ 825.364426] Key type ._llcrypt unregistered [ 828.737708] Key type ._llcrypt registered [ 828.738844] Key type .llcrypt registered [ 829.180765] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 07:51:28 (1784289088) [ 829.290873] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 829.296287] alg: No test for adler32 (adler32-zlib) [ 830.096882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 830.116167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 830.124684] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 830.352889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.13@tcp [ 830.366053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.201.13@tcp [ 830.374103] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 830.375743] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 830.377164] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 830, 0, 0 [ 830.379180] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 830.380330] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 830.382174] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 830.384123] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 830, 0, 0 [ 830.386969] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 830.388573] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 830.390229] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 830.391813] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 830, 0, 880 [ 830.394167] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 830.395385] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 830.397264] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 830.398877] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 830, 0, 880 [ 830.401146] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 832.363175] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 07:51:31 (1784289091) [ 832.525208] LNet: 107033:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 832.529242] LNetError: 107033:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 832.544430] LNet: Removed LNI 192.168.201.13@tcp [ 832.784430] Key type .llcrypt unregistered [ 832.786050] Key type ._llcrypt unregistered [ 834.731966] Key type ._llcrypt registered [ 834.733553] Key type .llcrypt registered [ 834.799122] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 834.804556] alg: No test for adler32 (adler32-zlib) [ 835.607380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 835.617130] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 838.390596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.113@tcp [ 840.936035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 840.949288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 840.957318] LNet: Added LNI 192.168.201.13@tcp1 [8/256/0/180] [ 841.254545] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 841.292701] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 841.294651] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 841.296468] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 843.432341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 843.440940] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 846.953178] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 07:51:45 (1784289105) [ 846.991894] LNet: 108530:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 846.994706] LNetError: 108530:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 847.000511] LNet: Removed LNI 192.168.201.13@tcp [ 848.031127] LNet: Removed LNI 192.168.201.13@tcp1 [ 848.032712] LNetError: 108530:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 848.035690] LNetError: 108530:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 848.044505] LNet: Removed LNI 10.1.2.3@tcp99 [ 848.279587] Key type .llcrypt unregistered [ 848.280755] Key type ._llcrypt unregistered [ 850.262342] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 850.267669] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 852.430233] Key type ._llcrypt registered [ 852.431407] Key type .llcrypt registered [ 852.491261] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 852.496319] alg: No test for adler32 (adler32-zlib) [ 853.300083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 853.323795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 853.333459] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 853.547877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 853.553960] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 854.212033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 854.226136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 854.242750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 854.252653] LNetError: 109279:0:(peer.c:2276:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 854.264889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 854.281382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.13@tcp [ 854.419724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 854.447558] LNet: 109299:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 854.449678] LNetError: 109299:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 854.451716] LNet: Removed LNI 10.1.2.3@tcp [ 855.462437] LNet: Removed LNI 192.168.201.13@tcp [ 855.680526] Key type .llcrypt unregistered [ 855.681837] Key type ._llcrypt unregistered [ 859.183138] Key type ._llcrypt registered [ 859.184635] Key type .llcrypt registered [ 859.645132] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 07:51:58 (1784289118) [ 859.760355] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 859.765870] alg: No test for adler32 (adler32-zlib) [ 860.573770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 860.597224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 860.607796] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 860.851180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.13@tcp [ 860.883025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg113-client@tcp [ 862.816249] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 07:52:01 (1784289121) [ 862.843594] LNet: 110823:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 862.846490] LNetError: 110823:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 862.857418] LNet: Removed LNI 192.168.201.13@tcp [ 863.054340] Key type .llcrypt unregistered [ 863.055274] Key type ._llcrypt unregistered [ 864.906288] Key type ._llcrypt registered [ 864.907818] Key type .llcrypt registered [ 864.957445] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 864.962572] alg: No test for adler32 (adler32-zlib) [ 865.756097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 865.764760] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 865.774531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 867.666856] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 07:52:06 (1784289126) [ 868.115765] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 868.648966] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 07:52:07 (1784289127) [ 869.156539] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 869.668947] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 07:52:08 (1784289128) [ 869.700246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 869.709565] LNet: 112241:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 869.712313] LNetError: 112241:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 869.719834] LNet: Removed LNI 192.168.201.13@tcp [ 869.729514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 869.750145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 869.759474] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 870.002210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 870.018536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 872.085475] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 07:52:10 (1784289130) [ 872.121320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 872.130495] LNet: 112951:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 872.133375] LNetError: 112951:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 872.144559] LNet: Removed LNI 192.168.201.13@tcp [ 872.156241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 872.177217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 872.189345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 872.401382] Key type .llcrypt unregistered [ 872.402554] Key type ._llcrypt unregistered [ 875.866904] Key type ._llcrypt registered [ 875.868527] Key type .llcrypt registered [ 876.278650] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 07:52:15 (1784289135) [ 876.382909] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 876.387087] alg: No test for adler32 (adler32-zlib) [ 877.181812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 877.201230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 877.211788] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 877.420859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 877.441338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.13@tcp [ 879.376411] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 07:52:18 (1784289138) [ 879.401274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 879.409283] LNet: 114502:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 879.411508] LNetError: 114502:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 879.416654] LNet: Removed LNI 192.168.201.13@tcp [ 879.424893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 879.445213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 879.455541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 881.395376] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 07:52:20 (1784289140) [ 881.421830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 881.432876] LNet: 115081:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 881.444733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 881.475260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 881.603381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 881.614887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 881.635268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 881.764940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 881.777329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 881.806054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 881.823636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 881.851591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 881.978252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 881.990218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 882.012190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 882.160282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 882.172460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 884.216915] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 07:52:23 (1784289143) [ 884.246599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 884.257488] LNet: 115716:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 884.259706] LNet: 115716:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 884.270614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 884.297958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 884.309202] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 884.577747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 884.588042] LNet: Added LNI 192.168.201.13@tcp2 [8/256/0/180] [ 884.911591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 884.935680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.13@tcp [ 885.020435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 885.028194] LNetError: 115999:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 885.030231] LNet: Removed LNI 192.168.201.13@tcp [ 885.036280] LNet: Removed LNI 192.168.201.13@tcp2 [ 885.043721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 885.066067] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 885.068203] LNet: Added LNI 192.168.201.13@tcp2 [8/256/0/180] [ 885.070047] LNet: 116005:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 885.240580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.13@tcp [ 885.315578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 886.367086] LNet: Removed LNI 192.168.201.13@tcp [ 886.368199] LNetError: 116042:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 886.370479] LNetError: 116042:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 886.380311] LNet: Removed LNI 192.168.201.13@tcp2 [ 886.387912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 886.412030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 886.422095] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 886.672807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 886.682368] LNet: Added LNI 192.168.201.13@tcp2 [8/256/0/180] [ 886.979586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.13@tcp [ 886.994252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 887.015178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.13@tcp [ 887.094514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 887.103718] LNet: 116326:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 887.105467] LNet: 116326:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 888.159139] LNet: Removed LNI 192.168.201.13@tcp [ 889.189317] LNet: Removed LNI 192.168.201.13@tcp2 [ 889.197412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 889.221836] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 889.224393] LNet: Added LNI 192.168.201.13@tcp2 [8/256/0/180] [ 889.226479] LNet: 116333:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 889.405908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.13@tcp [ 889.470741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 889.477539] LNetError: 116370:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 889.479488] LNetError: 116370:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 889.481443] LNet: Removed LNI 192.168.201.13@tcp [ 890.533308] LNet: Removed LNI 192.168.201.13@tcp2 [ 890.539913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 890.556634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 890.565496] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 890.794199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 890.802212] LNet: Added LNI 192.168.201.13@tcp2 [8/256/0/180] [ 891.055743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 891.075129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.13@tcp [ 891.142026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 891.149474] LNet: 116649:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 891.151709] LNet: 116649:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 892.191125] LNet: Removed LNI 192.168.201.13@tcp [ 892.198281] LNet: Removed LNI 192.168.201.13@tcp2 [ 892.204988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 892.226064] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 892.227799] LNet: Added LNI 192.168.201.13@tcp2 [8/256/0/180] [ 892.230361] LNet: 116655:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 892.376361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.13@tcp [ 892.440283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 893.471111] LNet: Removed LNI 192.168.201.13@tcp [ 893.478277] LNet: Removed LNI 192.168.201.13@tcp2 [ 893.485079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 893.503440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 893.512366] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 893.743918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 893.751177] LNet: Added LNI 192.168.201.13@tcp2 [8/256/0/180] [ 894.033481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.13@tcp [ 894.047446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 894.067474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.13@tcp [ 894.133458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 894.141336] LNetError: 116976:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 894.143524] LNetError: 116976:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 894.145741] LNet: Removed LNI 192.168.201.13@tcp [ 895.205254] LNet: Removed LNI 192.168.201.13@tcp2 [ 895.212542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 895.234187] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 895.236552] LNet: Added LNI 192.168.201.13@tcp2 [8/256/0/180] [ 895.238501] LNet: 116983:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 895.389155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.13@tcp [ 895.460259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 895.468349] LNet: Removed LNI 192.168.201.13@tcp [ 896.485304] LNet: Removed LNI 192.168.201.13@tcp2 [ 896.493525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 896.514775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 896.524643] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 896.767367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 896.775606] LNet: Added LNI 192.168.201.13@tcp2 [8/256/0/180] [ 897.051767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 897.063704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 897.085541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.13@tcp [ 897.205845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 898.271118] LNet: Removed LNI 192.168.201.13@tcp [ 898.278312] LNet: Removed LNI 192.168.201.13@tcp2 [ 898.285911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 898.305034] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 898.307296] LNet: Added LNI 192.168.201.13@tcp2 [8/256/0/180] [ 898.448350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.13@tcp [ 898.548822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 899.615116] LNet: Removed LNI 192.168.201.13@tcp [ 899.622275] LNet: Removed LNI 192.168.201.13@tcp2 [ 899.630233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 899.650825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 899.661092] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 899.897299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 899.907240] LNet: Added LNI 192.168.201.13@tcp2 [8/256/0/180] [ 900.176487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.13@tcp [ 900.191815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 900.203950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 900.223646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.13@tcp [ 900.338969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 900.346954] LNet: 117675:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 900.349658] LNet: 117675:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 901.407115] LNet: Removed LNI 192.168.201.13@tcp [ 902.437294] LNet: Removed LNI 192.168.201.13@tcp2 [ 902.445212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 902.463697] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 902.465392] LNet: Added LNI 192.168.201.13@tcp2 [8/256/0/180] [ 902.466972] LNet: 117682:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 902.469629] LNet: 117682:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 902.620119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.13@tcp [ 904.954896] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 905.479685] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 07:52:44 (1784289164) [ 905.543929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 905.554201] LNetError: 118441:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 905.557995] LNetError: 118441:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 9 previous similar messages [ 905.561566] LNet: Removed LNI 192.168.201.13@tcp [ 906.597511] LNet: Removed LNI 192.168.201.13@tcp2 [ 906.605738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 906.701647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 906.711285] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 906.961452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 906.970324] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 907.269865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 907.277784] LNet: Removed LNI 10.1.2.3@tcp [ 907.286279] LNet: Removed LNI 192.168.201.13@tcp [ 907.295161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 907.319085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 907.328905] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 907.330472] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 907.478187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 907.486964] LNet: Removed LNI 192.168.201.13@tcp [ 907.495887] LNet: Removed LNI 10.1.2.3@tcp [ 909.538294] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 07:52:48 (1784289168) [ 909.567596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 909.586477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 909.621624] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 909.657761] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 909.659548] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 909.661622] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 911.789864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 911.798180] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 912.040960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 912.051128] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 912.718777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.13@tcp [ 912.735264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 912.757581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.201.13@tcp -m PUT -e local_timeout -r 1 [ 912.779650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 912.803132] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 912.805568] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 912.881964] LNetError: 119381:0:(lib-move.c:4771:lnet_parse()) 192.168.201.13@tcp, src 192.168.201.13@tcp: Dropping PUT (error -108 looking up sender) [ 913.887623] LNet: Removed LNI 10.1.2.3@tcp [ 914.917308] LNet: Removed LNI 192.168.201.13@tcp [ 915.068796] Key type .llcrypt unregistered [ 915.069743] Key type ._llcrypt unregistered [ 918.479994] Key type ._llcrypt registered [ 918.480986] Key type .llcrypt registered [ 918.911874] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 07:52:57 (1784289177) [ 919.028177] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 919.033096] alg: No test for adler32 (adler32-zlib) [ 919.834577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 919.957756] LNet: 120690:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 920.185364] Key type .llcrypt unregistered [ 920.186637] Key type ._llcrypt unregistered [ 922.204817] Key type ._llcrypt registered [ 922.206098] Key type .llcrypt registered [ 922.278792] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 922.284673] alg: No test for adler32 (adler32-zlib) [ 923.086100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 923.097448] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 925.908732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.113@tcp [ 925.962573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 926.148451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.113@tcp [ 926.175063] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.113@tcp [ 926.225465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 926.238180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 926.254028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 926.269294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 926.283988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 926.298148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 926.312337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 926.326382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 926.335118] LNet: 121138:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 926.341136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 926.352722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 926.364276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 926.376181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 926.387485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 926.398235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 926.625739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 926.701660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 926.712914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 926.725549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 926.735661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 926.746573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 926.759222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 926.772533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 926.782904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 926.794435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 926.804798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 926.815564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 926.825324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 926.836890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 926.849944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 927.036496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.113@tcp [ 927.109703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 927.119268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 927.130325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 927.141201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 927.150868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 927.161244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 927.171366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 927.181759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 927.187722] LNet: 121372:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 927.190472] LNet: 121372:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 927.197163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 927.207634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 927.219736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 927.231769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 927.243835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 927.253478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 927.441428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 927.511809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 927.522470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 927.533461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 927.544087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 927.555457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 927.566542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 927.578792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 927.590124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 927.601088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 927.613349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 927.624241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 927.635222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 927.646385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 927.658883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 927.754910] LNet: 121535:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 927.757287] LNetError: 121535:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 927.772343] LNet: Removed LNI 192.168.201.13@tcp [ 927.904981] Key type .llcrypt unregistered [ 927.906020] Key type ._llcrypt unregistered [ 932.779573] Key type ._llcrypt registered [ 932.780540] Key type .llcrypt registered [ 933.181261] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 07:53:12 (1784289192) [ 933.288531] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 933.293745] alg: No test for adler32 (adler32-zlib) [ 934.090702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 934.204510] LNet: 122420:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 934.403255] Key type .llcrypt unregistered [ 934.404208] Key type ._llcrypt unregistered [ 936.185594] Key type ._llcrypt registered [ 936.186603] Key type .llcrypt registered [ 936.240953] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 936.245644] alg: No test for adler32 (adler32-zlib) [ 937.045248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 937.056179] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 939.740633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.113@tcp [ 942.092855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 942.105330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 942.112868] LNet: Added LNI 192.168.201.13@tcp1 [8/256/0/180] [ 942.382215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 942.561918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.113@tcp [ 942.637764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 942.649328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 942.660928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 942.671470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 942.682041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 942.691676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 942.700595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 942.710264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 942.716455] LNet: 123079:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 942.722410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 942.731225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 942.741105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 942.750572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 942.760212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 942.770809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 942.959946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 943.035635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 943.046485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 943.057635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 943.069158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 943.081280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 943.093174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 943.103914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 943.113707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 943.123900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 943.134492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 943.145623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 943.155253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 943.165870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 943.176200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 943.363451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.113@tcp [ 943.432014] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 943.442221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 943.452240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 943.461861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 943.471916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 943.481743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 943.491980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 943.501639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 943.508239] LNet: 123313:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 943.510586] LNet: 123313:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 943.517234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 943.527973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 943.539285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 943.549877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 943.560067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 943.571520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 943.761920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 943.834327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 943.846320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 943.857480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 943.871177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 943.882324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 943.893969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 943.905471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 943.916630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 943.928132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 943.940305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 943.950847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 943.960353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 943.970251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 943.979951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 945.978140] LNet: 123511:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 945.981077] LNetError: 123511:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 945.984360] LNet: Removed LNI 192.168.201.13@tcp [ 945.990319] LNet: Removed LNI 192.168.201.13@tcp1 [ 946.124750] Key type .llcrypt unregistered [ 946.125698] Key type ._llcrypt unregistered [ 951.133165] Key type ._llcrypt registered [ 951.134068] Key type .llcrypt registered [ 951.546416] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 07:53:30 (1784289210) [ 951.964566] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 952.458067] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 07:53:31 (1784289211) [ 954.521443] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 07:53:33 (1784289213) [ 956.468787] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 07:53:35 (1784289215) [ 958.495707] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 07:53:37 (1784289217) [ 960.670971] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 07:53:39 (1784289219) [ 962.781495] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 07:53:41 (1784289221) [ 964.905735] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 07:53:43 (1784289223) [ 967.256584] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 07:53:46 (1784289226) [ 969.345712] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 07:53:48 (1784289228) [ 971.416321] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 07:53:50 (1784289230) [ 973.406160] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 07:53:52 (1784289232) [ 975.594345] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 07:53:54 (1784289234) [ 977.695955] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 07:53:56 (1784289236) [ 979.686429] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 07:53:58 (1784289238) [ 981.740767] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 07:54:00 (1784289240) [ 983.814408] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 07:54:02 (1784289242) [ 985.877276] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 07:54:04 (1784289244) [ 987.907817] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 07:54:06 (1784289246) [ 988.014764] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 988.019710] alg: No test for adler32 (adler32-zlib) [ 988.816386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 988.852965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 988.860402] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 989.189552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 989.195946] LNet: 134500:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 989.198125] LNetError: 134500:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 989.205262] LNet: Removed LNI 192.168.201.13@tcp [ 989.213392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 989.230914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 989.239631] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 991.232928] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 07:54:10 (1784289250) [ 993.462617] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 07:54:12 (1784289252) [ 995.626201] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 07:54:14 (1784289254) [ 997.761916] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 07:54:16 (1784289256) [ 999.768641] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 07:54:18 (1784289258) [ 1001.861448] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 07:54:20 (1784289260) [ 1003.966482] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 07:54:22 (1784289262) [ 1006.187501] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 07:54:25 (1784289265) [ 1008.320649] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 07:54:27 (1784289267) [ 1010.410309] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 07:54:29 (1784289269) [ 1012.494876] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 07:54:31 (1784289271) [ 1014.638640] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 07:54:33 (1784289273) [ 1016.671189] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 893 sec =========== 07:54:35 (1784289275) [ 1016.754824] LNet: 141954:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1016.756872] LNetError: 141954:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1016.766501] LNet: Removed LNI 192.168.201.13@tcp [ 1016.981394] Key type .llcrypt unregistered [ 1016.982509] Key type ._llcrypt unregistered [ 1020.715183] Key type ._llcrypt registered [ 1020.721084] Key type .llcrypt registered [ 1020.838637] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1020.843442] alg: No test for adler32 (adler32-zlib) [ 1021.691136] Lustre: Lustre: Build Version: 2.17.54_173_g0a24909 [ 1021.770903] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 1023.351165] Key type lgssc registered [ 1023.693319] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1045.174202] Lustre: Mounted lustre-client [ 1047.596106] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1049.849156] Lustre: Unmounted lustre-client [ 1061.617152] Key type lgssc unregistered [ 1061.739458] LNet: 144868:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1061.743198] LNetError: 144868:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1061.751718] LNet: Removed LNI 192.168.201.13@tcp [ 1062.020127] Key type .llcrypt unregistered [ 1062.021350] Key type ._llcrypt unregistered