[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 450793621 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2421 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22BD 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 00227D (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2331 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23C1 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE23F9 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22bd-0xbffe2330] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22bc] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2331-0xbffe23c0] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23c1-0xbffe23f8] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe23f9-0xbffe2420] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2576MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524576K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.003108] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005016] kvm-guest: setup PV IPIs [ 0.007894] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009014] pid_max: default: 32768 minimum: 301 [ 0.010132] LSM: Security Framework initializing [ 0.011050] Yama: becoming mindful. [ 0.012037] SELinux: Initializing. [ 0.013072] *** VALIDATE selinux *** [ 0.021818] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026388] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027148] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028122] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029125] *** VALIDATE tmpfs *** [ 0.031352] *** VALIDATE proc *** [ 0.032240] *** VALIDATE cgroup *** [ 0.033008] *** VALIDATE cgroup2 *** [ 0.034265] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035149] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037030] Spectre V2 : User space: Vulnerable [ 0.038009] Speculative Store Bypass: Vulnerable [ 0.041030] debug: unmapping init [mem 0xffffffff9c459000-0xffffffff9c460fff] [ 0.043218] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044714] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045027] ... version: 2 [ 0.046013] ... bit width: 48 [ 0.047013] ... generic registers: 4 [ 0.048016] ... value mask: 0000ffffffffffff [ 0.049011] ... max period: 00007fffffffffff [ 0.050012] ... fixed-purpose events: 3 [ 0.051013] ... event mask: 000000070000000f [ 0.052341] rcu: Hierarchical SRCU implementation. [ 0.054559] smp: Bringing up secondary CPUs ... [ 0.055651] x86: Booting SMP configuration: [ 0.056033] .... node #0, CPUs: #1 #2 #3 [ 0.059567] smp: Brought up 1 node, 4 CPUs [ 0.061017] smpboot: Max logical packages: 1 [ 0.062024] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.218068] node 0 deferred pages initialised in 153ms [ 0.221428] devtmpfs: initialized [ 0.223520] x86/mm: Memory block size: 128MB [ 0.226321] gcov: version magic: 0x41383552 [ 0.230105] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.231094] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.232287] pinctrl core: initialized pinctrl subsystem [ 0.233213] [ 0.233885] ************************************************************* [ 0.234017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.235015] ** ** [ 0.236016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.237024] ** ** [ 0.238017] ** This means that this kernel is built to expose internal ** [ 0.239020] ** IOMMU data structures, which may compromise security on ** [ 0.240017] ** your system. ** [ 0.241017] ** ** [ 0.242016] ** If you see this message and you are not debugging the ** [ 0.243019] ** kernel, report this immediately to your vendor! ** [ 0.244025] ** ** [ 0.245088] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.246018] ************************************************************* [ 0.247750] NET: Registered protocol family 16 [ 0.250665] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.254090] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.256222] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.261015] cpuidle: using governor menu [ 0.262906] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.265752] PCI: Using configuration type 1 for base access [ 0.267169] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.277132] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.280109] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.284070] cryptd: max_cpu_qlen set to 1000 [ 0.287273] ACPI: Added _OSI(Module Device) [ 0.288020] ACPI: Added _OSI(Processor Device) [ 0.290019] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.291018] ACPI: Added _OSI(Processor Aggregator Device) [ 0.296403] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.302399] ACPI: Interpreter enabled [ 0.303066] ACPI: PM: (supports S0 S3 S4 S5) [ 0.305015] ACPI: Using IOAPIC for interrupt routing [ 0.306148] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.309435] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.320253] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.322044] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.324024] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.327122] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.332705] acpiphp: Slot [2] registered [ 0.334340] acpiphp: Slot [5] registered [ 0.336169] acpiphp: Slot [6] registered [ 0.337147] acpiphp: Slot [3] registered [ 0.338138] acpiphp: Slot [4] registered [ 0.339108] acpiphp: Slot [7] registered [ 0.341329] acpiphp: Slot [8] registered [ 0.342133] acpiphp: Slot [9] registered [ 0.343121] acpiphp: Slot [10] registered [ 0.345105] acpiphp: Slot [11] registered [ 0.346128] acpiphp: Slot [12] registered [ 0.347105] acpiphp: Slot [13] registered [ 0.348087] acpiphp: Slot [14] registered [ 0.349141] acpiphp: Slot [15] registered [ 0.351137] acpiphp: Slot [16] registered [ 0.352123] acpiphp: Slot [17] registered [ 0.353100] acpiphp: Slot [18] registered [ 0.354109] acpiphp: Slot [19] registered [ 0.356089] acpiphp: Slot [20] registered [ 0.357090] acpiphp: Slot [21] registered [ 0.358108] acpiphp: Slot [22] registered [ 0.359123] acpiphp: Slot [23] registered [ 0.361111] acpiphp: Slot [24] registered [ 0.362113] acpiphp: Slot [25] registered [ 0.364131] acpiphp: Slot [26] registered [ 0.365123] acpiphp: Slot [27] registered [ 0.367270] acpiphp: Slot [28] registered [ 0.368135] acpiphp: Slot [29] registered [ 0.370388] acpiphp: Slot [30] registered [ 0.371148] acpiphp: Slot [31] registered [ 0.373076] PCI host bridge to bus 0000:00 [ 0.374026] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.376029] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.378028] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.380032] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.382033] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.385033] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.386249] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.390078] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.393636] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.402018] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.406067] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.409025] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.411021] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.413023] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.414737] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.417773] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.420044] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.421871] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.426018] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.433025] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.437652] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.443846] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.451022] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.461027] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.474020] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.483036] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.489023] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.494022] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.505024] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.514229] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.516403] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.519542] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.521382] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.524254] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.528241] iommu: Default domain type: Passthrough [ 0.530142] SCSI subsystem initialized [ 0.531652] ACPI: bus type USB registered [ 0.532000] usbcore: registered new interface driver usbfs [ 0.535115] usbcore: registered new interface driver hub [ 0.537204] usbcore: registered new device driver usb [ 0.539299] pps_core: LinuxPPS API ver. 1 registered [ 0.541014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.544094] PTP clock support registered [ 0.546160] EDAC MC: Ver: 3.0.0 [ 0.548161] PCI: Using ACPI for IRQ routing [ 0.550828] NetLabel: Initializing [ 0.552014] NetLabel: domain hash size = 128 [ 0.553010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.555118] NetLabel: unlabeled traffic allowed by default [ 0.557208] vgaarb: loaded [ 0.559458] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.560015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.566997] clocksource: Switched to clocksource kvm-clock [ 0.692809] VFS: Disk quotas dquot_6.6.0 [ 0.694418] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.696661] *** VALIDATE ramfs *** [ 0.697858] *** VALIDATE hugetlbfs *** [ 0.699522] pnp: PnP ACPI init [ 0.701995] pnp: PnP ACPI: found 6 devices [ 0.718631] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.721492] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.723244] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.725192] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.727493] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.729713] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.732389] NET: Registered protocol family 2 [ 0.736421] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.741732] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.745776] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.751269] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.755398] TCP: Hash tables configured (established 65536 bind 65536) [ 0.758613] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.762604] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.766273] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.769812] NET: Registered protocol family 1 [ 0.772841] RPC: Registered named UNIX socket transport module. [ 0.774744] RPC: Registered udp transport module. [ 0.776079] RPC: Registered tcp transport module. [ 0.777516] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.779342] NET: Registered protocol family 44 [ 0.780878] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.782853] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.784664] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.786459] PCI: CLS 0 bytes, default 64 [ 0.787778] Unpacking initramfs... [ 2.213941] debug: unmapping init [mem 0xffff8d86fcc64000-0xffff8d86fffcffff] [ 2.220689] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.223886] software IO TLB: mapped [mem 0x00000000b8c64000-0x00000000bcc64000] (64MB) [ 2.227761] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.724418] Initialise system trusted keyrings [ 2.725570] Key type blacklist registered [ 2.726861] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.734102] zbud: loaded [ 2.736423] *** VALIDATE nfs *** [ 2.737391] *** VALIDATE nfs4 *** [ 2.738598] pstore: using deflate compression [ 2.741672] Platform Keyring initialized [ 2.848328] NET: Registered protocol family 38 [ 2.850540] Key type asymmetric registered [ 2.852385] Asymmetric key parser 'x509' registered [ 2.854760] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.858868] io scheduler mq-deadline registered [ 2.862096] io scheduler kyber registered [ 2.863677] io scheduler bfq registered [ 2.866093] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.868405] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.871449] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.874511] ACPI: Power Button [PWRF] [ 2.880064] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.887112] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.908843] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.935018] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.963946] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.967615] Non-volatile memory driver v1.3 [ 2.968768] Linux agpgart interface v0.103 [ 2.995247] virtio_blk virtio1: [vda] 134720 512-byte logical blocks (69.0 MB/65.8 MiB) [ 2.998664] vda: detected capacity change from 0 to 68976640 [ 3.020621] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.023266] vdb: detected capacity change from 0 to 1073741824 [ 3.030098] libphy: Fixed MDIO Bus: probed [ 3.047285] usbcore: registered new interface driver usbserial_generic [ 3.056412] usbserial: USB Serial support registered for generic [ 3.058485] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.062147] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.064119] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.068692] mousedev: PS/2 mouse device common for all mice [ 3.073723] rtc_cmos 00:05: RTC can wake from S4 [ 3.077164] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.078930] rtc_cmos 00:05: registered as rtc0 [ 3.083335] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.089624] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.090706] intel_pstate: CPU model not supported [ 3.100575] hid: raw HID events driver (C) Jiri Kosina [ 3.102441] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.105420] usbcore: registered new interface driver usbhid [ 3.107136] usbhid: USB HID core driver [ 3.108246] drop_monitor: Initializing network drop monitor service [ 3.110079] Initializing XFRM netlink socket [ 3.111608] NET: Registered protocol family 10 [ 3.113892] Segment Routing with IPv6 [ 3.115658] NET: Registered protocol family 17 [ 3.117749] mpls_gso: MPLS GSO support [ 3.123389] RAS: Correctable Errors collector initialized. [ 3.125610] AVX version of gcm_enc/dec engaged. [ 3.127334] AES CTR mode by8 optimization enabled [ 3.216679] sched_clock: Marking stable (3216655778, 0)->(4098652210, -881996432) [ 3.220428] registered taskstats version 1 [ 3.223732] Loading compiled-in X.509 certificates [ 3.226294] zswap: loaded using pool lzo/zbud [ 3.252861] Key type big_key registered [ 3.266666] Key type encrypted registered [ 3.268287] ima: No TPM chip found, activating TPM-bypass! [ 3.270620] ima: Allocated hash algorithm: sha1 [ 3.272494] ima: No architecture policies found [ 3.274539] evm: Initialising EVM extended attributes: [ 3.276683] evm: security.selinux [ 3.278186] evm: security.ima [ 3.279341] evm: security.capability [ 3.280785] evm: HMAC attrs: 0x1 [ 3.283267] rtc_cmos 00:05: setting system clock to 2026-03-02 17:02:21 UTC (1772470941) [ 3.290268] debug: unmapping init [mem 0xffffffff9d403000-0xffffffff9d5fffff] [ 3.293447] debug: unmapping init [mem 0xffffffff9c182000-0xffffffff9c458fff] [ 3.302348] Write protecting the kernel read-only data: 28672k [ 3.306550] debug: unmapping init [mem 0xffffffff9a803000-0xffffffff9a9fffff] [ 3.309770] debug: unmapping init [mem 0xffffffff9b114000-0xffffffff9b1fffff] [ 3.345179] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.353849] systemd[1]: Detected virtualization kvm. [ 3.355864] systemd[1]: Detected architecture x86-64. [ 3.357773] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.381522] systemd[1]: No hostname configured. [ 3.383494] systemd[1]: Set hostname to . [ 3.385879] random: systemd: uninitialized urandom read (16 bytes read) [ 3.388728] systemd[1]: Initializing machine ID from random generator. [ 3.429235] random: ln: uninitialized urandom read (6 bytes read) [ 3.555707] random: systemd: uninitialized urandom read (16 bytes read) [ 3.558280] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.565267] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.569152] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.221637] device-mapper: uevent: version 1.0.3 [ 4.223716] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.063725] random: fast init done [ 5.119075] scsi host0: ata_piix [ 5.127485] virtio_net virtio0 ens2: renamed from eth0 [ 5.211166] scsi host1: ata_piix [ 5.212805] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.215410] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.053461] dracut-initqueue[586]: RTNETLINK answers: File exists [ 9.785337] random: crng init done [ 9.786727] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.331657] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.498538] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.733878] SELinux: Disabled at runtime. [ 11.792286] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.801717] systemd[1]: Detected virtualization kvm. [ 11.803377] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.264276] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.267752] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.272845] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.278301] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.281859] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.289192] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.300453] systemd[1]: Listening on initctl Compatibility Named Pipe. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Kernel Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Mounting Huge Pages File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Control Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on RPCbind Server Activation Socket. Starting udev Coldplug all Devices... [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-getty.slice. Mounting POSIX Message Queue File System... Mounting Kernel Debug File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. Activating swap /dev/disk/by-label/SWAP... Starting Remount Root and Kernel File Systems... [ OK ] Reached target Local Encrypted Volumes. [ 12.504119] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Apply Kernel Variables... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.775915] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.039811] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.086128] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.178866] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.188117] EDAC sbridge: Ver: 1.1.2 [ 14.304639] Key type dns_resolver registered [ 14.615254] NFS: Registering the id_resolver key type [ 14.617229] Key type id_resolver registered [ 14.618353] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Login Service... Starting Network Manager... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Started Authorization Manager. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg156-client login: [ 52.442868] libcfs: loading out-of-tree module taints kernel. [ 52.595913] Key type ._llcrypt registered [ 52.637503] Key type .llcrypt registered [ 53.231496] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 53.241846] alg: No test for adler32 (adler32-zlib) [ 54.903728] Lustre: Lustre: Build Version: 2.17.50_196_gee1d501 [ 56.033507] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 57.751153] Key type lgssc registered [ 60.097194] Lustre: Echo OBD driver; http://www.lustre.org/ [ 174.081241] hrtimer: interrupt took 6287851 ns [ 194.236662] Lustre: Mounted lustre-client [ 199.328618] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 220.060420] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing check_logdir /tmp/testlogs/ [ 220.127990] Lustre: lustre-OST0000-osc-ffff8d87487a0800: disconnect after 24s idle [ 225.143796] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing yml_node [ 229.281202] Lustre: DEBUG MARKER: Client: 2.17.50.196 [ 231.676277] Lustre: DEBUG MARKER: MDS: 2.17.50.196 [ 233.854267] Lustre: DEBUG MARKER: OSS: 2.17.50.196 [ 235.903105] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Mon Mar 2 12:06:12 EST 2026 [ 252.113830] LustreError: 7931:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff8d87487a0800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 252.140974] Lustre: Unmounted lustre-client [ 270.059448] Key type lgssc unregistered [ 270.390967] LNet: 8380:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 270.413428] LNetError: 8380:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 270.427840] LNet: Removed LNI 192.168.201.56@tcp [ 271.238270] Key type .llcrypt unregistered [ 271.239469] Key type ._llcrypt unregistered [ 295.155146] Key type ._llcrypt registered [ 295.156742] Key type .llcrypt registered [ 295.407740] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 295.437558] alg: No test for adler32 (adler32-zlib) [ 296.357937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 296.416473] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 296.459629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 300.915414] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 300.998088] LNet: 9251:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 301.010571] LNetError: 9251:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 301.038642] LNet: Removed LNI 192.168.201.56@tcp [ 301.625572] Key type .llcrypt unregistered [ 301.627450] Key type ._llcrypt unregistered [ 308.977710] Key type ._llcrypt registered [ 308.988442] Key type .llcrypt registered [ 310.142663] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 12:07:27 (1772471247) [ 310.566293] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 310.589289] alg: No test for adler32 (adler32-zlib) [ 311.533706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 311.586451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 311.601454] LNet: 9670:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 311.613378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 311.646236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 311.659881] LNet: 9677:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 318.038533] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 12:07:35 (1772471255) [ 318.368394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 318.408855] LNet: 10254:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 318.433714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 318.523931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 324.614662] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 12:07:41 (1772471261) [ 324.835386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 324.857442] LNet: 10839:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 324.875579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 324.941498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 331.524873] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 12:07:48 (1772471268) [ 331.627640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 331.659549] LNet: 11420:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 331.688258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 331.780385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 338.326883] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 12:07:55 (1772471275) [ 338.641994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 338.676669] LNet: 12009:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 338.718712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 338.835718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 339.363651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 347.940860] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 12:08:04 (1772471284) [ 348.350806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 348.390816] LNet: 12602:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 348.431945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 348.556813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 349.273840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 349.781801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 355.601729] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 12:08:12 (1772471292) [ 355.797253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 355.875677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 355.966121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 362.724319] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 12:08:19 (1772471299) [ 363.046538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 363.095565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 363.173489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 363.207359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 363.648318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 363.701490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 364.261787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 364.308818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 364.780758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 364.825614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 365.245756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 365.305484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 365.796770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 365.844201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 366.445825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 366.470587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 366.927533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 366.973383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 367.373293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 367.445514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 367.849814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 367.882388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 368.274384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 368.345935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 368.881195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 368.921397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 369.308617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 369.355480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 376.337026] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 12:08:33 (1772471313) [ 376.468631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 376.510219] LNet: 14463:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 376.528626] LNet: 14463:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 376.588566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 376.671252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 376.720021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 383.265583] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 12:08:40 (1772471320) [ 383.386726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 383.439246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 383.503861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 383.526351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 389.279506] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 12:08:46 (1772471326) [ 389.392371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 389.457787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 389.628106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 389.668633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 398.443796] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 12:08:55 (1772471335) [ 398.567165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 398.674881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 398.784613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 398.826513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 406.200621] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 12:09:02 (1772471342) [ 406.305505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 406.381398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 406.488268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 406.516410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 412.764671] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 12:09:09 (1772471349) [ 413.143128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 413.172499] LNet: 17398:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 413.177143] LNet: 17398:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 413.226108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 413.314993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 413.383438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 419.924618] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 12:09:16 (1772471356) [ 420.078951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 420.128056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 420.250111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 420.683461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 426.423618] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 12:09:23 (1772471363) [ 426.494722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 426.541366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 426.780142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 426.985593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 434.006236] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 12:09:31 (1772471371) [ 434.164736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 434.194729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 434.291996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 434.470706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 441.438632] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 12:09:38 (1772471378) [ 441.561617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 441.632692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 442.257204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 442.445422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 449.753517] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 12:09:46 (1772471386) [ 449.940434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 449.973607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 450.042120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 450.097733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 457.870101] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 12:09:54 (1772471394) [ 458.079629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 458.122147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 458.254593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 458.318628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 467.498145] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 12:10:03 (1772471403) [ 467.689598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 467.783261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 467.848353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 474.691604] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 12:10:11 (1772471411) [ 474.841462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 474.901643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 474.991613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 481.393600] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 12:10:18 (1772471418) [ 481.503856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 481.540671] LNet: 22829:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 481.550665] LNet: 22829:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 481.604271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 481.762480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 482.218300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 489.101672] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 12:10:25 (1772471425) [ 489.222940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 489.303567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 489.427314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 489.459747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 496.161664] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 12:10:32 (1772471432) [ 496.319607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 496.374719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 496.485976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 496.547318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 505.048763] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 12:10:41 (1772471441) [ 505.272412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 505.371036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 505.461057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 505.530734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 514.486403] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 12:10:51 (1772471451) [ 514.687490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 514.723185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 514.780302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 514.836591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 521.440325] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 12:10:58 (1772471458) [ 521.699647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 521.739230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 521.810769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 521.910606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 521.929562] LNetError: 25785:0:(peer.c:2165:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 521.963278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 521.993480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 529.940524] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 12:11:06 (1772471466) [ 530.322888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 530.424082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 530.530868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.aWH8VeUbsX [ 530.713510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.aWH8VeUbsX [ 530.806621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.aWH8VeUbsX [ 538.017619] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 12:11:14 (1772471474) [ 538.383770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 538.457127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 538.528476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 538.556901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 538.649127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 545.928711] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 12:11:22 (1772471482) [ 546.118530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 546.232337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 546.333839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 551.673072] LNet: Message forwarding will be enabled in 2s [ 554.783233] LNet: Message forwarding enabled [ 555.681154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 555.780559] LNet: Message forwarding will stop in 2s to 6s [ 559.482026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 561.184918] LNet: Message forwarding disabled [ 562.651850] LNet: Message forwarding will be enabled in 2s [ 566.303274] LNet: Message forwarding enabled [ 570.422926] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 12:11:47 (1772471507) [ 570.529723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 572.377704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 572.443608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 580.020923] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 12:11:56 (1772471516) [ 580.148298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 580.198261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 580.269229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 580.300671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 580.327291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 580.352433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 580.375529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 580.402643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 580.424341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 580.449738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 580.479956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 580.502624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 580.523449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 580.549566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 580.574426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 580.616424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 580.643699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 580.673715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 580.705385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 580.731813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 580.764801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 580.789399] LNetError: 28848:0:(router.c:624:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 587.359506] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 12:12:04 (1772471524) [ 587.466253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 587.548665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 587.648702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 594.466843] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 12:12:11 (1772471531) [ 596.738813] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg156-server [ 598.911895] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 12:12:15 (1772471535) [ 599.073396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 599.132193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 599.196708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 599.216524] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 600.943887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.201.[57-61]@tcp [ 600.968522] LNet: 30565:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 601.013096] LNetError: 30355:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.58@tcp is being used as a gateway but routing feature is not turned on [ 601.036121] LNetError: 30355:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) route to tcp1 through 192.168.201.58@tcp has gone from up to down [ 602.027967] LNetError: 30355:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.57@tcp is being used as a gateway but routing feature is not turned on [ 602.050899] LNetError: 30355:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 7 previous similar messages [ 603.056342] LNetError: 30355:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.58@tcp is being used as a gateway but routing feature is not turned on [ 603.065505] LNetError: 30355:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 3 previous similar messages [ 604.127994] LNetError: 30360:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) route to tcp1 through 192.168.201.61@tcp has gone from up to down [ 604.142475] LNetError: 30360:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) Skipped 3 previous similar messages [ 605.109836] LNetError: 30355:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.59@tcp is being used as a gateway but routing feature is not turned on [ 605.129528] LNetError: 30355:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 7 previous similar messages [ 608.415599] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 12:12:24 (1772471544) [ 609.195895] LNetError: 30355:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.57@tcp is being used as a gateway but routing feature is not turned on [ 609.207895] LNetError: 30355:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 15 previous similar messages [ 610.512903] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg156-server [ 612.508535] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 12:12:29 (1772471549) [ 612.745305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 612.785896] LNet: 31500:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 612.797504] LNet: 31500:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 12 previous similar messages [ 612.820820] LNetError: 31500:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 612.872333] LNet: Removed LNI 192.168.201.56@tcp [ 612.935773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 613.023467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 613.076898] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 613.830437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.201.[57-62/2]@tcp [ 613.841955] LNet: 31617:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 613.846236] LNet: 31617:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 613.868833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.201.[57-62/2]@tcp [ 613.883844] LNetError: 31504:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) route to tcp103 through 192.168.201.59@tcp has gone from up to down [ 619.855282] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 12:12:36 (1772471556) [ 619.994867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 620.041696] LNetError: 32209:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 620.063706] LNet: Removed LNI 192.168.201.56@tcp [ 620.080725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 620.151961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 620.201665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 620.233368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 620.282603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 620.327471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 620.357836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 620.397669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 620.484360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 620.532972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 620.601307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 620.643545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 620.693160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 620.745583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 620.836735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 620.880410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 620.948342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 621.035631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 621.086251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 621.182306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 621.269963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 621.311606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 621.354710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 621.404632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 621.430233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 621.470352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 621.542985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 621.567303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 621.610921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 621.672573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 621.695891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 621.733889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 621.775361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 621.814945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 621.880691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 628.040573] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 12:12:44 (1772471564) [ 630.019964] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg156-server [ 631.773198] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 12:12:48 (1772471568) [ 633.724814] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg156-server [ 635.177529] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 12:12:52 (1772471572) [ 636.203359] Key type .llcrypt unregistered [ 636.207471] Key type ._llcrypt unregistered [ 642.198728] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 642.381986] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 642.385674] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 642.388622] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 645.049417] Key type ._llcrypt registered [ 645.052063] Key type .llcrypt registered [ 645.270459] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 645.283941] alg: No test for adler32 (adler32-zlib) [ 646.295991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 646.354289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 646.383926] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 647.304990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 647.318311] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 648.995591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 649.017366] LNet: Removed LNI 10.1.2.3@tcp [ 649.125758] LNet: 34332:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 649.133250] LNetError: 34332:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 649.142609] LNet: Removed LNI 192.168.201.56@tcp [ 649.692755] Key type .llcrypt unregistered [ 649.695565] Key type ._llcrypt unregistered [ 660.020062] Key type ._llcrypt registered [ 660.024404] Key type .llcrypt registered [ 661.169355] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 12:13:18 (1772471598) [ 661.787724] Key type .llcrypt unregistered [ 661.790069] Key type ._llcrypt unregistered [ 667.835390] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 667.998643] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 668.014841] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 668.029276] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 670.654342] Key type ._llcrypt registered [ 670.655787] Key type .llcrypt registered [ 670.829255] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 670.849853] alg: No test for adler32 (adler32-zlib) [ 671.863986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 671.949183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 671.983895] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 672.738667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 672.767323] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 675.390620] LNet: 35828:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 675.395565] LNetError: 35828:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 675.400598] LNet: Removed LNI 192.168.201.56@tcp [ 675.422362] LNet: Removed LNI 10.1.2.3@tcp [ 676.089172] Key type .llcrypt unregistered [ 676.096551] Key type ._llcrypt unregistered [ 687.108568] Key type ._llcrypt registered [ 687.110507] Key type .llcrypt registered [ 688.364654] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 12:13:45 (1772471625) [ 688.932364] Key type .llcrypt unregistered [ 688.934257] Key type ._llcrypt unregistered [ 694.126457] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 694.262973] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 694.265353] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 694.280409] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 696.911951] Key type ._llcrypt registered [ 696.914573] Key type .llcrypt registered [ 697.147374] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 697.160528] alg: No test for adler32 (adler32-zlib) [ 698.102463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 698.172861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 698.202672] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 699.668434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 699.687624] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 701.791781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 701.826757] LNet: Removed LNI 10.1.2.3@tcp [ 701.838996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 701.880992] LNet: Removed LNI 10.1.2.31@tcp [ 701.897570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 701.929162] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 702.780381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 702.795319] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 703.723611] LNet: 37743:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 703.727830] LNetError: 37743:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 703.733108] LNet: Removed LNI 10.1.2.3@tcp [ 703.744256] LNet: Removed LNI 10.1.2.31@tcp [ 704.400506] Key type .llcrypt unregistered [ 704.404974] Key type ._llcrypt unregistered [ 716.164497] Key type ._llcrypt registered [ 716.167270] Key type .llcrypt registered [ 717.491228] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 12:14:14 (1772471654) [ 717.914681] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 717.923952] alg: No test for adler32 (adler32-zlib) [ 718.785045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 718.831744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 718.856986] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 719.548073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 719.582459] LNet: 38698:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 719.589723] LNetError: 38698:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 719.608437] LNet: Removed LNI 192.168.201.56@tcp [ 719.623207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 719.688984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.201.56@tcp [ 719.709360] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 720.450084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.201.56@tcp [ 720.501703] LNet: Removed LNI 192.168.201.56@tcp [ 720.565356] LNet: 38834:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 721.229860] Key type .llcrypt unregistered [ 721.238191] Key type ._llcrypt unregistered [ 733.795442] Key type ._llcrypt registered [ 733.798802] Key type .llcrypt registered [ 735.338535] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 12:14:31 (1772471671) [ 737.135225] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg156-server [ 739.480813] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 12:14:35 (1772471675) [ 739.980361] Key type .llcrypt unregistered [ 739.986949] Key type ._llcrypt unregistered [ 745.677854] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 745.821112] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 745.828245] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 745.834861] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 748.420598] Key type ._llcrypt registered [ 748.425574] Key type .llcrypt registered [ 748.603630] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 748.614453] alg: No test for adler32 (adler32-zlib) [ 749.584561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 749.668807] LNet: Added LNI 192.168.201.56@tcp1 [8/256/0/180] [ 750.528541] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 752.692351] LNet: 40673:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 752.697732] LNetError: 40673:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 752.702425] LNet: Removed LNI 192.168.201.56@tcp1 [ 752.711842] LNet: Removed LNI 10.1.2.3@tcp2 [ 753.390584] Key type .llcrypt unregistered [ 753.397190] Key type ._llcrypt unregistered [ 766.211666] Key type ._llcrypt registered [ 766.215838] Key type .llcrypt registered [ 767.570978] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 12:15:04 (1772471704) [ 768.212843] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 768.237445] alg: No test for adler32 (adler32-zlib) [ 769.194840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 769.282977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 769.340850] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 770.330122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 770.352506] LNet: 41642:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 770.363846] LNetError: 41642:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 770.377681] LNet: Removed LNI 192.168.201.56@tcp [ 770.397263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 770.472254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 770.499167] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 776.838865] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 12:15:13 (1772471713) [ 777.054472] LNet: 42233:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 777.067861] LNetError: 42233:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 777.091729] LNet: Removed LNI 192.168.201.56@tcp [ 777.783984] Key type .llcrypt unregistered [ 777.789166] Key type ._llcrypt unregistered [ 784.639843] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 784.825265] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 784.840912] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 784.852901] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 787.524570] Key type ._llcrypt registered [ 787.525919] Key type .llcrypt registered [ 787.712779] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 787.733816] alg: No test for adler32 (adler32-zlib) [ 788.646690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 788.721554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 788.737651] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 789.411544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 789.442801] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 791.805189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 791.863563] LNet: 42971:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 791.873727] LNetError: 42971:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 791.888297] LNet: Removed LNI 192.168.201.56@tcp [ 791.902276] LNet: Removed LNI 10.1.2.3@tcp [ 791.949297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 792.035551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 792.066765] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 792.071582] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 792.866856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 792.887453] LNet: 42998:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 792.892889] LNetError: 42998:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 792.901378] LNetError: 42998:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 792.905969] LNet: Removed LNI 192.168.201.56@tcp [ 792.925589] LNet: Removed LNI 10.1.2.3@tcp [ 792.952765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 793.039118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 793.063376] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 793.067196] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 793.697560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 793.722679] LNet: Added LNI 192.168.201.56@tcp2 [8/256/0/180] [ 794.735110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 794.754115] LNet: 43160:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 794.759755] LNetError: 43160:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 794.764124] LNetError: 43160:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 794.769292] LNet: Removed LNI 192.168.201.56@tcp [ 794.771527] LNet: Removed LNI 10.1.2.3@tcp [ 794.778663] LNet: Removed LNI 192.168.201.56@tcp2 [ 794.803324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 794.867469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 794.888229] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 794.891440] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 794.894865] LNet: Added LNI 192.168.201.56@tcp2 [8/256/0/180] [ 802.903669] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 12:15:39 (1772471739) [ 803.261689] LNet: 43759:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 803.265939] LNetError: 43759:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 803.279907] LNetError: 43759:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 803.298679] LNet: Removed LNI 192.168.201.56@tcp [ 803.300612] LNet: Removed LNI 10.1.2.3@tcp [ 803.329791] LNet: Removed LNI 192.168.201.56@tcp2 [ 804.178705] Key type .llcrypt unregistered [ 804.184825] Key type ._llcrypt unregistered [ 810.266870] Key type ._llcrypt registered [ 810.269858] Key type .llcrypt registered [ 810.670864] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 810.693212] alg: No test for adler32 (adler32-zlib) [ 811.659211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 811.742450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 811.755681] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 812.522473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 812.543371] LNet: 44153:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 812.553439] LNetError: 44153:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 812.561416] LNet: Removed LNI 192.168.201.56@tcp [ 812.581520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 812.637600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 812.665768] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 813.265940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 813.315765] LNet: 44184:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 813.328599] LNetError: 44184:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 813.343621] LNet: Removed LNI 192.168.201.56@tcp [ 813.373244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 813.435929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 813.458860] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 814.001920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 814.049555] LNet: Removed LNI 192.168.201.56@tcp [ 814.066906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 814.124936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 814.152174] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 814.665763] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 814.979096] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 814.992416] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 815.009303] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 817.537558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 817.562046] LNet: 44292:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 817.570603] LNet: 44292:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 817.590411] LNetError: 44292:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 817.596216] LNetError: 44292:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 817.621974] LNet: Removed LNI 192.168.201.56@tcp [ 817.643584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 817.723639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 817.760742] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 825.035977] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 12:16:01 (1772471761) [ 825.130563] LNet: 44882:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 825.137657] LNetError: 44882:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 825.147790] LNet: Removed LNI 192.168.201.56@tcp [ 826.051861] Key type .llcrypt unregistered [ 826.054207] Key type ._llcrypt unregistered [ 833.059587] Key type ._llcrypt registered [ 833.062194] Key type .llcrypt registered [ 833.215368] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 833.236136] alg: No test for adler32 (adler32-zlib) [ 834.293434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 834.362871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 834.400988] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 835.485601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 835.550046] LNet: 45290:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 835.563907] LNetError: 45290:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 835.584943] LNet: Removed LNI 192.168.201.56@tcp [ 835.645207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 835.769724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 835.807847] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 842.856280] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 12:16:19 (1772471779) [ 843.143274] LNet: 45882:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 843.151125] LNetError: 45882:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 843.167828] LNet: Removed LNI 192.168.201.56@tcp [ 843.856486] Key type .llcrypt unregistered [ 843.858242] Key type ._llcrypt unregistered [ 850.652486] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 850.986663] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 850.992617] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 850.999938] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 853.716772] Key type ._llcrypt registered [ 853.722373] Key type .llcrypt registered [ 853.908384] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 853.924919] alg: No test for adler32 (adler32-zlib) [ 854.889138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 854.962458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 854.989912] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 855.958684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 856.012591] LNet: 46333:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 856.020966] LNetError: 46333:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 856.037456] LNet: Removed LNI 192.168.201.56@tcp [ 856.065776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 856.128655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 856.146596] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 863.434336] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 12:16:40 (1772471800) [ 863.583919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 863.621754] LNet: 46926:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 863.631824] LNetError: 46926:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 863.657157] LNet: Removed LNI 192.168.201.56@tcp [ 863.705451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 863.855357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 870.368381] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 12:16:47 (1772471807) [ 870.476995] LNet: 47506:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 871.138582] Key type .llcrypt unregistered [ 871.146507] Key type ._llcrypt unregistered [ 878.165493] Key type ._llcrypt registered [ 878.170673] Key type .llcrypt registered [ 878.393795] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 878.418540] alg: No test for adler32 (adler32-zlib) [ 879.355774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 879.430340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 879.487241] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 880.624316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 880.656160] LNet: 47911:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 880.660921] LNetError: 47911:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 880.680622] LNet: Removed LNI 192.168.201.56@tcp [ 880.694211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 880.748153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 880.773967] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 888.071467] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 12:17:04 (1772471824) [ 888.208863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 888.252221] LNet: 48497:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 888.275928] LNetError: 48497:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 888.303456] LNet: Removed LNI 192.168.201.56@tcp [ 888.377508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 888.487213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 894.965502] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 12:17:11 (1772471831) [ 896.875560] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 899.133708] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 12:17:15 (1772471835) [ 900.707187] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 902.681453] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 12:17:19 (1772471839) [ 904.348480] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 906.275175] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 12:17:22 (1772471842) [ 908.238945] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 909.750687] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 12:17:26 (1772471846) [ 911.278752] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 913.200227] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 12:17:30 (1772471850) [ 913.499729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 913.521774] LNet: 50559:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 913.540593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 913.590540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 913.614914] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 914.373987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.201.56 [ 914.386742] LNet: 50692:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 914.401199] LNet: Removed LNI 192.168.201.56@tcp [ 921.880353] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 12:17:38 (1772471858) [ 922.042552] LNet: 51254:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 922.064471] LNetError: 51254:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 922.107957] LNet: Removed LNI 192.168.201.56@tcp [ 922.697712] Key type .llcrypt unregistered [ 922.698985] Key type ._llcrypt unregistered [ 929.460638] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 929.899680] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 929.914403] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 929.920357] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 931.626874] Key type ._llcrypt registered [ 931.633865] Key type .llcrypt registered [ 931.898181] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 931.916238] alg: No test for adler32 (adler32-zlib) [ 933.465741] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 941.369431] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 12:17:58 (1772471878) [ 941.588689] LNet: 52156:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 941.593900] LNetError: 52156:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 941.619784] LNet: Removed LNI 10.1.2.3@tcp [ 942.373366] Key type .llcrypt unregistered [ 942.378898] Key type ._llcrypt unregistered [ 949.551097] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 949.851095] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 949.860644] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 949.870406] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 952.666354] Key type ._llcrypt registered [ 952.668261] Key type .llcrypt registered [ 952.821701] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 952.832269] alg: No test for adler32 (adler32-zlib) [ 953.867607] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 960.980166] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 12:18:17 (1772471897) [ 961.220544] LNet: 53058:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 961.227188] LNetError: 53058:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 961.246267] LNet: Removed LNI 10.1.2.3@tcp [ 961.996676] Key type .llcrypt unregistered [ 961.998179] Key type ._llcrypt unregistered [ 968.808117] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 969.016182] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 969.021135] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 969.030974] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 971.762782] Key type ._llcrypt registered [ 971.766786] Key type .llcrypt registered [ 972.062335] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 972.083193] alg: No test for adler32 (adler32-zlib) [ 972.972596] LNet: 53359:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 972.984182] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 980.362589] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 12:18:36 (1772471916) [ 980.493078] LNet: 53944:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 980.508778] LNetError: 53944:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 980.542769] LNet: Removed LNI 10.1.2.3@tcp [ 981.308865] Key type .llcrypt unregistered [ 981.311888] Key type ._llcrypt unregistered [ 987.650632] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 987.862350] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 987.865882] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 987.872770] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 990.727256] Key type ._llcrypt registered [ 990.733164] Key type .llcrypt registered [ 991.034431] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 991.070470] alg: No test for adler32 (adler32-zlib) [ 992.034763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 992.111232] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1001.370401] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 12:18:58 (1772471938) [ 1002.505056] LNet: 55182:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1002.511301] LNetError: 55182:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1002.536652] LNet: Removed LNI 10.1.2.3@tcp [ 1003.363171] Key type .llcrypt unregistered [ 1003.366125] Key type ._llcrypt unregistered [ 1010.582863] Key type ._llcrypt registered [ 1010.590615] Key type .llcrypt registered [ 1010.999501] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1011.028218] alg: No test for adler32 (adler32-zlib) [ 1012.007091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1012.040372] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 1020.534909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.156@tcp [ 1021.317657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.56@tcp -d 192.168.201.156@tcp -m GET -e local_interrupt -r 1 [ 1021.364522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.56@tcp -d 192.168.201.56@tcp -m GET -e local_interrupt -r 1 [ 1021.423441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.156@tcp [ 1021.462753] LNet: There was an unexpected network error while writing to 192.168.201.156: rc = -22 [ 1021.765773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1021.816932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1021.867571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1021.910342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1021.966569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1022.010800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1022.093938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1022.155215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1022.183510] LNet: 55624:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1022.223158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1022.261490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1022.303723] LNet: 1 local NIs in recovery (showing 1): 192.168.201.56@tcp [ 1022.308213] LNet: There was an unexpected network error while writing to 192.168.201.56: rc = -22 [ 1022.325724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1022.400083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1022.439542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1022.488492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1022.536979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1023.189749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.56@tcp -d 192.168.201.156@tcp -m GET -e local_dropped -r 1 [ 1023.217729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.56@tcp -d 192.168.201.56@tcp -m GET -e local_dropped -r 1 [ 1023.249687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.156@tcp [ 1023.475897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1023.503649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1023.533868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1023.571944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1023.620773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1023.644903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1023.676552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1023.709555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1023.738468] LNet: 55734:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1023.751908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1023.787090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1023.828594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1023.858878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1023.896933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1023.934406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1023.968432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1024.627939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.56@tcp -d 192.168.201.156@tcp -m GET -e local_aborted -r 1 [ 1024.661447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.56@tcp -d 192.168.201.56@tcp -m GET -e local_aborted -r 1 [ 1024.688037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.156@tcp [ 1024.705442] LNet: There was an unexpected network error while writing to 192.168.201.156: rc = -22 [ 1024.710507] LNet: Skipped 1 previous similar message [ 1024.944990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1024.976798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1025.009974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1025.040140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1025.064993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1025.092156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1025.115202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1025.141874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1025.160933] LNet: 55843:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1025.181478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1025.206520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1025.233650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1025.268503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1025.305840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1025.333687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1025.389126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1026.084088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.56@tcp -d 192.168.201.156@tcp -m GET -e local_no_route -r 1 [ 1026.169037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.56@tcp -d 192.168.201.56@tcp -m GET -e local_no_route -r 1 [ 1026.298965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.156@tcp [ 1026.785723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1026.835769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1026.868840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1026.931459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1026.987530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1027.068592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1027.122191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1027.148377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1027.185937] LNet: 55952:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1027.232075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1027.286387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1027.351576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1027.414933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1027.509091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1027.594469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1027.690076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1028.247469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.56@tcp -d 192.168.201.156@tcp -m GET -e local_timeout -r 1 [ 1028.296372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.56@tcp -d 192.168.201.56@tcp -m GET -e local_timeout -r 1 [ 1028.356972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.156@tcp [ 1028.384511] LNet: There was an unexpected network error while writing to 192.168.201.156: rc = -22 [ 1028.387813] LNet: Skipped 1 previous similar message [ 1028.707194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1028.811380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1028.873947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1028.942739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1029.010390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1029.072268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1029.121245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1029.172751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1029.236233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1029.275608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1029.299718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1029.334619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1029.370287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1029.399396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1029.425540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1030.333476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.56@tcp -d 192.168.201.156@tcp -m GET -e local_error -r 1 [ 1030.420634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.56@tcp -d 192.168.201.56@tcp -m GET -e local_error -r 1 [ 1030.499878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.156@tcp [ 1030.763620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1030.832578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1030.886314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1030.923582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1030.978973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1031.059592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1031.104306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1031.145983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1031.189380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1031.238325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1031.271770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1031.331802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1031.389875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1031.455308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1031.577581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1032.060466] LNet: 56212:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1032.064474] LNetError: 56212:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1032.080589] LNet: Removed LNI 192.168.201.56@tcp [ 1032.576503] Key type .llcrypt unregistered [ 1032.578220] Key type ._llcrypt unregistered [ 1053.995673] Key type ._llcrypt registered [ 1053.997562] Key type .llcrypt registered [ 1055.134120] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 12:19:52 (1772471992) [ 1055.947505] Key type .llcrypt unregistered [ 1055.950814] Key type ._llcrypt unregistered [ 1062.055195] Key type ._llcrypt registered [ 1062.056755] Key type .llcrypt registered [ 1062.369304] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1062.394311] alg: No test for adler32 (adler32-zlib) [ 1063.420184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1063.464405] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 1071.889722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.156@tcp [ 1083.884397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1083.992944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1084.034480] LNet: Added LNI 192.168.201.56@tcp1 [8/256/0/180] [ 1086.640694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.56@tcp -d 192.168.201.156@tcp -m GET -e local_interrupt -r 1 [ 1086.700216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.56@tcp -d 192.168.201.156@tcp1 -m GET -e local_interrupt -r 1 [ 1086.765624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.56@tcp -d 192.168.201.56@tcp -m GET -e local_interrupt -r 1 [ 1086.840800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.56@tcp -d 192.168.201.56@tcp1 -m GET -e local_interrupt -r 1 [ 1086.905840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.56@tcp1 -d 192.168.201.156@tcp -m GET -e local_interrupt -r 1 [ 1086.972847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.56@tcp1 -d 192.168.201.156@tcp1 -m GET -e local_interrupt -r 1 [ 1087.010567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.56@tcp1 -d 192.168.201.56@tcp -m GET -e local_interrupt -r 1 [ 1087.068564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.56@tcp1 -d 192.168.201.56@tcp1 -m GET -e local_interrupt -r 1 [ 1087.117714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.156@tcp [ 1087.156788] LNet: There was an unexpected network error while writing to 192.168.201.156: rc = -22 [ 1087.170748] LNet: 1 local NIs in recovery (showing 1): 192.168.201.56@tcp [ 1087.541393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1087.582912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1087.610519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1087.643926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1087.677824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1087.713159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1087.743844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1087.783785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1087.801760] LNet: 57679:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1087.825643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1087.857763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1087.881607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1087.899629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1087.924423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1087.955397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1087.977341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1088.693647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.56@tcp -d 192.168.201.156@tcp -m GET -e local_dropped -r 1 [ 1088.728454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.56@tcp -d 192.168.201.156@tcp1 -m GET -e local_dropped -r 1 [ 1088.759412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.56@tcp -d 192.168.201.56@tcp -m GET -e local_dropped -r 1 [ 1088.787231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.56@tcp -d 192.168.201.56@tcp1 -m GET -e local_dropped -r 1 [ 1088.815541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.56@tcp1 -d 192.168.201.156@tcp -m GET -e local_dropped -r 1 [ 1088.837634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.56@tcp1 -d 192.168.201.156@tcp1 -m GET -e local_dropped -r 1 [ 1088.862413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.56@tcp1 -d 192.168.201.56@tcp -m GET -e local_dropped -r 1 [ 1088.889264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.56@tcp1 -d 192.168.201.56@tcp1 -m GET -e local_dropped -r 1 [ 1088.920982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.156@tcp [ 1088.934403] LNet: There was an unexpected network error while writing to 192.168.201.156: rc = -22 [ 1088.940366] LNet: Skipped 2 previous similar messages [ 1089.258750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1089.310330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1089.365657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1089.433637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1089.479191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1089.567677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1089.627514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1089.654153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1089.698550] LNet: 57801:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1089.747475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1089.804711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1089.841147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1089.881342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1089.914631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1089.965817] LNet: There was an unexpected network error while writing to 192.168.201.56: rc = -22 [ 1089.969532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1089.975530] LNet: Skipped 3 previous similar messages [ 1090.016519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1090.673193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.56@tcp -d 192.168.201.156@tcp -m GET -e local_aborted -r 1 [ 1090.716945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.56@tcp -d 192.168.201.156@tcp1 -m GET -e local_aborted -r 1 [ 1090.764534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.56@tcp -d 192.168.201.56@tcp -m GET -e local_aborted -r 1 [ 1090.840936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.56@tcp -d 192.168.201.56@tcp1 -m GET -e local_aborted -r 1 [ 1090.888869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.56@tcp1 -d 192.168.201.156@tcp -m GET -e local_aborted -r 1 [ 1090.950980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.56@tcp1 -d 192.168.201.156@tcp1 -m GET -e local_aborted -r 1 [ 1090.982951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.56@tcp1 -d 192.168.201.56@tcp -m GET -e local_aborted -r 1 [ 1091.043902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.56@tcp1 -d 192.168.201.56@tcp1 -m GET -e local_aborted -r 1 [ 1091.085704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.156@tcp [ 1091.316959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1091.358577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1091.386173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1091.421640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1091.448652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1091.471294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1091.503306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1091.544551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1091.569559] LNet: 57922:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1091.586644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1091.662664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1091.698601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1091.747832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1091.783580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1091.833503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1091.866347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1092.437516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.56@tcp -d 192.168.201.156@tcp -m GET -e local_no_route -r 1 [ 1092.489912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.56@tcp -d 192.168.201.156@tcp1 -m GET -e local_no_route -r 1 [ 1092.551502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.56@tcp -d 192.168.201.56@tcp -m GET -e local_no_route -r 1 [ 1092.599102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.56@tcp -d 192.168.201.56@tcp1 -m GET -e local_no_route -r 1 [ 1092.670807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.56@tcp1 -d 192.168.201.156@tcp -m GET -e local_no_route -r 1 [ 1092.725819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.56@tcp1 -d 192.168.201.156@tcp1 -m GET -e local_no_route -r 1 [ 1092.782893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.56@tcp1 -d 192.168.201.56@tcp -m GET -e local_no_route -r 1 [ 1092.847873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.56@tcp1 -d 192.168.201.56@tcp1 -m GET -e local_no_route -r 1 [ 1092.888086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.156@tcp [ 1092.927768] LNet: There was an unexpected network error while writing to 192.168.201.156: rc = -22 [ 1092.933946] LNet: Skipped 3 previous similar messages [ 1093.183688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1093.213916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1093.248306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1093.267789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1093.299917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1093.356226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1093.391613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1093.427930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1093.478260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1093.522911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1093.548686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1093.575938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1093.612462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1093.646302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1093.683560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1094.357911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.56@tcp -d 192.168.201.156@tcp -m GET -e local_timeout -r 1 [ 1094.417328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.56@tcp -d 192.168.201.156@tcp1 -m GET -e local_timeout -r 1 [ 1094.478517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.56@tcp -d 192.168.201.56@tcp -m GET -e local_timeout -r 1 [ 1094.516480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.56@tcp -d 192.168.201.56@tcp1 -m GET -e local_timeout -r 1 [ 1094.569679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.56@tcp1 -d 192.168.201.156@tcp -m GET -e local_timeout -r 1 [ 1094.621326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.56@tcp1 -d 192.168.201.156@tcp1 -m GET -e local_timeout -r 1 [ 1094.650979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.56@tcp1 -d 192.168.201.56@tcp -m GET -e local_timeout -r 1 [ 1094.729262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.56@tcp1 -d 192.168.201.56@tcp1 -m GET -e local_timeout -r 1 [ 1094.792575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.156@tcp [ 1095.070496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1095.120620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1095.158447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1095.196564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1095.218157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1095.266736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1095.314890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1095.356869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1095.403199] LNet: 58165:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1095.410371] LNet: 58165:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1095.431087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1095.487559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1095.537132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1095.573585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1095.600085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1095.669853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1095.738961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1096.358702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.56@tcp -d 192.168.201.156@tcp -m GET -e local_error -r 1 [ 1096.391537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.56@tcp -d 192.168.201.156@tcp1 -m GET -e local_error -r 1 [ 1096.427569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.56@tcp -d 192.168.201.56@tcp -m GET -e local_error -r 1 [ 1096.463459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.56@tcp -d 192.168.201.56@tcp1 -m GET -e local_error -r 1 [ 1096.506364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.56@tcp1 -d 192.168.201.156@tcp -m GET -e local_error -r 1 [ 1096.552565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.56@tcp1 -d 192.168.201.156@tcp1 -m GET -e local_error -r 1 [ 1096.590292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.56@tcp1 -d 192.168.201.56@tcp -m GET -e local_error -r 1 [ 1096.626466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.56@tcp1 -d 192.168.201.56@tcp1 -m GET -e local_error -r 1 [ 1096.648154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.156@tcp [ 1096.909762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1096.936272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1096.979549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1097.014073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1097.051151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1097.087441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1097.126251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1097.159641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1097.210745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1097.282268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1097.326092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1097.358983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1097.390210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1097.439661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1097.498209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1103.598739] LNet: 58362:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1103.603580] LNetError: 58362:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1103.613486] LNet: Removed LNI 192.168.201.56@tcp [ 1104.679266] LNet: Removed LNI 192.168.201.56@tcp1 [ 1105.246798] Key type .llcrypt unregistered [ 1105.248426] Key type ._llcrypt unregistered [ 1122.145987] Key type ._llcrypt registered [ 1122.150779] Key type .llcrypt registered [ 1123.433024] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 12:21:00 (1772472060) [ 1124.078575] Key type .llcrypt unregistered [ 1124.086251] Key type ._llcrypt unregistered [ 1130.565561] Key type ._llcrypt registered [ 1130.573218] Key type .llcrypt registered [ 1130.805758] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1130.856118] alg: No test for adler32 (adler32-zlib) [ 1131.748588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1131.780323] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 1139.270588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.156@tcp [ 1139.949660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.56@tcp -d 192.168.201.156@tcp -m GET -e remote_dropped -r 1 [ 1139.985430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.56@tcp -d 192.168.201.56@tcp -m GET -e remote_dropped -r 1 [ 1140.020685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.156@tcp [ 1140.067693] LNet: There was an unexpected network error while writing to 192.168.201.156: rc = -22 [ 1140.353567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1140.388254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1140.430996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1140.473092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1140.517552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1140.555759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1140.606318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1140.663829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1140.695403] LNet: 59612:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1140.718678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1140.754319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1140.798922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1140.831852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1140.860810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1140.892822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1140.931505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1141.423331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.56@tcp -d 192.168.201.156@tcp -m GET -e remote_error -r 1 [ 1141.453202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.56@tcp -d 192.168.201.56@tcp -m GET -e remote_error -r 1 [ 1141.497941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.156@tcp [ 1141.533594] LNet: There was an unexpected network error while writing to 192.168.201.156: rc = -22 [ 1141.766854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1141.793978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1141.825224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1141.864957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1141.897986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1141.928075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1141.966790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1142.002849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1142.034973] LNet: 59721:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1142.070936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1142.100570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1142.158537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1142.188229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1142.247373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1142.291846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1142.320479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1142.952735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.56@tcp -d 192.168.201.156@tcp -m GET -e remote_timeout -r 1 [ 1142.983587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.56@tcp -d 192.168.201.56@tcp -m GET -e remote_timeout -r 1 [ 1143.014598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.156@tcp [ 1143.028825] LNet: There was an unexpected network error while writing to 192.168.201.156: rc = -22 [ 1143.204696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1143.233259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1143.257669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1143.283772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1143.306675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1143.341566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1143.373658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1143.408361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1143.426199] LNet: 59830:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1143.439608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1143.461281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1143.489889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1143.515060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1143.551832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1143.579443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1143.609552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1143.808399] LNet: 59871:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1143.813293] LNetError: 59871:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1143.831353] LNet: Removed LNI 192.168.201.56@tcp [ 1144.151573] Key type .llcrypt unregistered [ 1144.154508] Key type ._llcrypt unregistered [ 1160.321522] Key type ._llcrypt registered [ 1160.324387] Key type .llcrypt registered [ 1161.535197] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 12:21:38 (1772472098) [ 1162.450914] Key type .llcrypt unregistered [ 1162.454406] Key type ._llcrypt unregistered [ 1169.048489] Key type ._llcrypt registered [ 1169.052524] Key type .llcrypt registered [ 1169.323492] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1169.342996] alg: No test for adler32 (adler32-zlib) [ 1170.275879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1170.331925] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 1177.463400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.156@tcp [ 1186.844946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1186.935943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1187.018207] LNet: Added LNI 192.168.201.56@tcp1 [8/256/0/180] [ 1188.675431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.56@tcp -d 192.168.201.156@tcp -m GET -e remote_dropped -r 1 [ 1188.718863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.56@tcp -d 192.168.201.156@tcp1 -m GET -e remote_dropped -r 1 [ 1188.779696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.56@tcp -d 192.168.201.56@tcp -m GET -e remote_dropped -r 1 [ 1188.839655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.56@tcp -d 192.168.201.56@tcp1 -m GET -e remote_dropped -r 1 [ 1188.878065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.56@tcp1 -d 192.168.201.156@tcp -m GET -e remote_dropped -r 1 [ 1188.910613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.56@tcp1 -d 192.168.201.156@tcp1 -m GET -e remote_dropped -r 1 [ 1188.949753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.56@tcp1 -d 192.168.201.56@tcp -m GET -e remote_dropped -r 1 [ 1189.005931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.56@tcp1 -d 192.168.201.56@tcp1 -m GET -e remote_dropped -r 1 [ 1189.075243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.156@tcp [ 1189.107880] LNet: There was an unexpected network error while writing to 192.168.201.156: rc = -22 [ 1189.113427] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.156@tcp [ 1189.387587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1189.416370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1189.444272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1189.505110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1189.560977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1189.612466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1189.666710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1189.749879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1189.782436] LNet: 61340:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1189.800186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1189.835829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1189.875631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1189.912162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1189.949553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1189.993656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1190.046909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1190.076827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1190.646193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.56@tcp -d 192.168.201.156@tcp -m GET -e remote_error -r 1 [ 1190.673279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.56@tcp -d 192.168.201.156@tcp1 -m GET -e remote_error -r 1 [ 1190.700917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.56@tcp -d 192.168.201.56@tcp -m GET -e remote_error -r 1 [ 1190.730922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.56@tcp -d 192.168.201.56@tcp1 -m GET -e remote_error -r 1 [ 1190.762182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.56@tcp1 -d 192.168.201.156@tcp -m GET -e remote_error -r 1 [ 1190.789380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.56@tcp1 -d 192.168.201.156@tcp1 -m GET -e remote_error -r 1 [ 1190.821115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.56@tcp1 -d 192.168.201.56@tcp -m GET -e remote_error -r 1 [ 1190.863200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.56@tcp1 -d 192.168.201.56@tcp1 -m GET -e remote_error -r 1 [ 1190.890747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.156@tcp [ 1190.910155] LNet: There was an unexpected network error while writing to 192.168.201.156: rc = -22 [ 1190.914663] LNet: Skipped 2 previous similar messages [ 1191.082455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1191.102476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1191.124755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1191.148390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1191.171240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1191.199947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1191.225579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1191.254692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1191.269860] LNet: 61464:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1191.297852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1191.337618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1191.376439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1191.428872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1191.468437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1191.521414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1191.591729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1191.650474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1192.377352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.56@tcp -d 192.168.201.156@tcp -m GET -e remote_timeout -r 1 [ 1192.445483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.56@tcp -d 192.168.201.156@tcp1 -m GET -e remote_timeout -r 1 [ 1192.510795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.56@tcp -d 192.168.201.56@tcp -m GET -e remote_timeout -r 1 [ 1192.596135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.56@tcp -d 192.168.201.56@tcp1 -m GET -e remote_timeout -r 1 [ 1192.705716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.56@tcp1 -d 192.168.201.156@tcp -m GET -e remote_timeout -r 1 [ 1192.784189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.56@tcp1 -d 192.168.201.156@tcp1 -m GET -e remote_timeout -r 1 [ 1192.857554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.56@tcp1 -d 192.168.201.56@tcp -m GET -e remote_timeout -r 1 [ 1192.913771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.56@tcp1 -d 192.168.201.56@tcp1 -m GET -e remote_timeout -r 1 [ 1192.967139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.156@tcp [ 1193.009685] LNet: There was an unexpected network error while writing to 192.168.201.156: rc = -22 [ 1193.015917] LNet: Skipped 1 previous similar message [ 1193.320514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1193.363203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1193.439529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1193.469426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1193.505473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1193.529514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1193.563934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1193.594452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1193.613827] LNet: 61587:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1193.630158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1193.658636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1193.686598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1193.710812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1193.743955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1193.766145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1193.787474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1193.847468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1198.630733] LNet: 61665:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1198.638559] LNetError: 61665:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1198.651236] LNet: Removed LNI 192.168.201.56@tcp [ 1198.665933] LNet: Removed LNI 192.168.201.56@tcp1 [ 1199.054702] Key type .llcrypt unregistered [ 1199.059299] Key type ._llcrypt unregistered [ 1215.568383] Key type ._llcrypt registered [ 1215.574663] Key type .llcrypt registered [ 1216.847597] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 12:22:33 (1772472153) [ 1217.307417] Key type .llcrypt unregistered [ 1217.309038] Key type ._llcrypt unregistered [ 1223.222863] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1223.339832] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1223.342945] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1223.349562] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1226.035456] Key type ._llcrypt registered [ 1226.048621] Key type .llcrypt registered [ 1226.262441] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1226.271595] alg: No test for adler32 (adler32-zlib) [ 1227.210616] LNet: 62742:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1227.219218] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 1227.248213] LNet: 62757:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1227.253498] LNetError: 62757:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1227.267685] LNet: Removed LNI 192.168.201.56@tcp [ 1228.242499] Key type .llcrypt unregistered [ 1228.249843] Key type ._llcrypt unregistered [ 1234.753511] Key type ._llcrypt registered [ 1234.755706] Key type .llcrypt registered [ 1234.945490] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1234.964729] alg: No test for adler32 (adler32-zlib) [ 1235.959903] LNet: 63016:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1235.977488] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 1235.984754] LNet: Added LNI 10.1.2.3@tcp1 [8/256/0/180] [ 1236.053503] LNet: 63030:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1236.065649] LNetError: 63030:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1236.080870] LNet: Removed LNI 192.168.201.56@tcp [ 1236.105248] LNet: Removed LNI 10.1.2.3@tcp1 [ 1236.733412] Key type .llcrypt unregistered [ 1236.738872] Key type ._llcrypt unregistered [ 1242.572305] Key type ._llcrypt registered [ 1242.574860] Key type .llcrypt registered [ 1242.751269] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1242.786590] alg: No test for adler32 (adler32-zlib) [ 1243.664828] LNet: 63289:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1243.675762] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 1243.677529] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1243.709792] LNet: 63303:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1243.713904] LNetError: 63303:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1243.717862] LNet: Removed LNI 192.168.201.56@tcp [ 1243.726595] LNet: Removed LNI 10.1.2.3@tcp [ 1244.264348] Key type .llcrypt unregistered [ 1244.268812] Key type ._llcrypt unregistered [ 1249.778982] Key type ._llcrypt registered [ 1249.781525] Key type .llcrypt registered [ 1249.986201] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1250.016195] alg: No test for adler32 (adler32-zlib) [ 1250.931775] LNet: 63563:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1250.944049] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 1250.946621] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1250.984800] LNet: 63577:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1250.992302] LNetError: 63577:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1250.997756] LNet: Removed LNI 192.168.201.56@tcp [ 1251.007462] LNet: Removed LNI 10.1.2.3@tcp [ 1251.563509] Key type .llcrypt unregistered [ 1251.572430] Key type ._llcrypt unregistered [ 1257.764658] Key type ._llcrypt registered [ 1257.768659] Key type .llcrypt registered [ 1257.959099] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1257.985484] alg: No test for adler32 (adler32-zlib) [ 1258.903761] LNet: 63842:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1258.913020] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 1258.915201] LNetError: 63842:0:(config.c:1593:lnet_inet_select()) ksocklnd: failed to find UP interface test1pg [ 1258.919721] LNetError: Error -100 starting up LNI tcp [ 1258.930325] LNet: Removed LNI 192.168.201.56@tcp [ 1259.598509] Key type .llcrypt unregistered [ 1259.600704] Key type ._llcrypt unregistered [ 1271.259800] Key type ._llcrypt registered [ 1271.261612] Key type .llcrypt registered [ 1272.928804] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 12:23:29 (1772472209) [ 1274.129629] Key type .llcrypt unregistered [ 1274.134370] Key type ._llcrypt unregistered [ 1280.739833] Key type ._llcrypt registered [ 1280.743829] Key type .llcrypt registered [ 1281.140907] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1281.173349] alg: No test for adler32 (adler32-zlib) [ 1282.252882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1282.295059] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 1291.375921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.156@tcp [ 1292.126399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.56@tcp -d 192.168.201.156@tcp -m GET -e network_timeout -r 1 [ 1292.169419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.56@tcp -d 192.168.201.56@tcp -m GET -e network_timeout -r 1 [ 1292.210688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.156@tcp [ 1292.234278] LNet: There was an unexpected network error while writing to 192.168.201.156: rc = -22 [ 1292.455033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1292.489650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1292.530406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1292.562840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1292.594956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1292.625195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1292.645114] LNet: 1 local NIs in recovery (showing 1): 192.168.201.56@tcp [ 1292.667588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1292.710659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1292.749851] LNet: 65099:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1292.797928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1292.856602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1292.918265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1292.956417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1293.040730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1293.111866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1293.431552] LNet: 65138:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1293.441317] LNetError: 65138:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1293.484419] LNet: Removed LNI 192.168.201.56@tcp [ 1294.069577] Key type .llcrypt unregistered [ 1294.077500] Key type ._llcrypt unregistered [ 1314.887954] Key type ._llcrypt registered [ 1314.890160] Key type .llcrypt registered [ 1315.115452] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1315.179990] alg: No test for adler32 (adler32-zlib) [ 1316.164767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1316.196491] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 1324.191853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.156@tcp [ 1332.859676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1332.903743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1332.942441] LNet: Added LNI 192.168.201.56@tcp1 [8/256/0/180] [ 1334.344694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.56@tcp -d 192.168.201.156@tcp -m GET -e network_timeout -r 1 [ 1334.381963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.56@tcp -d 192.168.201.156@tcp1 -m GET -e network_timeout -r 1 [ 1334.415272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.56@tcp -d 192.168.201.56@tcp -m GET -e network_timeout -r 1 [ 1334.470726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.56@tcp -d 192.168.201.56@tcp1 -m GET -e network_timeout -r 1 [ 1334.528536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.56@tcp1 -d 192.168.201.156@tcp -m GET -e network_timeout -r 1 [ 1334.580590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.56@tcp1 -d 192.168.201.156@tcp1 -m GET -e network_timeout -r 1 [ 1334.608578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.56@tcp1 -d 192.168.201.56@tcp -m GET -e network_timeout -r 1 [ 1334.659716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.56@tcp1 -d 192.168.201.56@tcp1 -m GET -e network_timeout -r 1 [ 1334.693940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.156@tcp [ 1334.716663] LNet: There was an unexpected network error while writing to 192.168.201.156: rc = -22 [ 1334.957444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1334.986471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1335.009476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1335.037974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1335.077481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1335.133576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1335.195303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1335.231479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1335.248853] LNet: 66020:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1335.266324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1335.291324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1335.319135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1335.351206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1335.390928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1335.424730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1339.791106] LNet: There was an unexpected network error while writing to 192.168.201.156: rc = -22 [ 1340.128490] LNet: 66094:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1340.135380] LNetError: 66094:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1341.151206] LNet: Removed LNI 192.168.201.56@tcp [ 1341.155069] LNetError: 66094:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1341.170631] LNet: Removed LNI 192.168.201.56@tcp1 [ 1341.765888] Key type .llcrypt unregistered [ 1341.768092] Key type ._llcrypt unregistered [ 1357.119747] Key type ._llcrypt registered [ 1357.121983] Key type .llcrypt registered [ 1358.597302] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 12:24:55 (1772472295) [ 1359.037752] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1359.048206] alg: No test for adler32 (adler32-zlib) [ 1360.004295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1360.077523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1360.108054] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 1360.857968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1360.884124] LNet: Added LNI 192.168.201.56@tcp1 [8/256/0/180] [ 1361.966871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.56@tcp [ 1362.054210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1362.100458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1362.136770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1362.194854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1362.264934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1362.314235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.56@tcp [ 1362.343699] LNetError: 67195:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.201.56@tcp to recovery [ 1363.106785] LNet: 1 local NIs in recovery (showing 1): 192.168.201.56@tcp [ 1363.108719] LNet: There was an unexpected network error while writing to 192.168.201.56: rc = -22 [ 1365.152060] LNet: There was an unexpected network error while writing to 192.168.201.56: rc = -22 [ 1366.255173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1369.250104] LNet: There was an unexpected network error while writing to 192.168.201.56: rc = -22 [ 1369.988890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1370.045615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1370.123431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1370.144628] LNet: 67302:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1370.153678] LNetError: 67302:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1371.169284] LNet: Removed LNI 192.168.201.56@tcp [ 1371.176078] LNetError: 67302:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1371.212850] LNet: Removed LNI 192.168.201.56@tcp1 [ 1371.277127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1371.391871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1371.438840] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 1372.396979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1372.445572] LNet: Added LNI 192.168.201.56@tcp1 [8/256/0/180] [ 1373.537811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.56@tcp [ 1373.591908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1373.662093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1373.752801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1373.803691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1373.845669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1373.879464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1373.942886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.56@tcp [ 1373.968536] LNetError: 67575:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.201.56@tcp to recovery [ 1374.367753] LNet: There was an unexpected network error while writing to 192.168.201.56: rc = -22 [ 1377.680800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1380.512649] LNet: There was an unexpected network error while writing to 192.168.201.56: rc = -22 [ 1380.524858] LNet: Skipped 1 previous similar message [ 1384.958617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1385.016786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1385.056799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1391.759749] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 12:25:28 (1772472328) [ 1391.882348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1391.902239] LNet: 68276:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1391.919921] LNetError: 68276:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1391.939858] LNet: Removed LNI 192.168.201.56@tcp [ 1391.962981] LNet: Removed LNI 192.168.201.56@tcp1 [ 1392.029638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1392.104734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1392.139159] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 1392.721048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1392.740473] LNet: Added LNI 192.168.201.56@tcp1 [8/256/0/180] [ 1393.518412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.56@tcp [ 1393.571333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1393.596832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1393.620254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1393.650247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1393.669403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1393.698931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.56@tcp --health 0 [ 1393.786153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1394.144485] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.56@tcp [ 1394.146450] LNet: There was an unexpected network error while writing to 192.168.201.56: rc = -22 [ 1394.167545] LNet: Skipped 1 previous similar message [ 1408.956023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1409.213986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1409.265364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1409.317214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.56@tcp --health 500 [ 1409.456966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1416.661150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1416.774217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1416.809940] LNet: 68767:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1416.817129] LNetError: 68767:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1416.830420] LNetError: 68767:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1416.840571] LNet: Removed LNI 192.168.201.56@tcp [ 1416.870771] LNet: Removed LNI 192.168.201.56@tcp1 [ 1416.925721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1417.004281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1417.026130] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 1418.249820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1418.319841] LNet: Added LNI 192.168.201.56@tcp1 [8/256/0/180] [ 1419.391119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.56@tcp [ 1419.469815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1419.552565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1419.600541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1419.649784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1419.690690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1419.729612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1419.785304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.56@tcp --health 0 [ 1420.063530] LNet: There was an unexpected network error while writing to 192.168.201.56: rc = -22 [ 1420.066417] LNet: Skipped 3 previous similar messages [ 1421.245343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1430.616477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1430.681522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1430.745397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1438.037534] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 12:26:15 (1772472375) [ 1438.618534] LNet: 69801:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1438.623883] LNetError: 69801:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1438.628315] LNetError: 69801:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1438.633352] LNet: Removed LNI 192.168.201.56@tcp [ 1438.645781] LNet: Removed LNI 192.168.201.56@tcp1 [ 1439.273307] Key type .llcrypt unregistered [ 1439.275289] Key type ._llcrypt unregistered [ 1446.149841] Key type ._llcrypt registered [ 1446.151887] Key type .llcrypt registered [ 1446.381886] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1446.404171] alg: No test for adler32 (adler32-zlib) [ 1447.303337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1447.328961] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 1456.221637] Lustre: DEBUG MARKER: Initial discovery [ 1456.362552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.156@tcp [ 1459.010436] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1459.126242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1459.177193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.156@tcp [ 1459.218394] LNet: There was an unexpected network error while writing to 192.168.201.156: rc = -22 [ 1459.257783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1459.289547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.201.156@tcp -r 1 -m GET -l 3 [ 1459.319820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.201.156@tcp -r 1 -m GET -e local_error [ 1459.349904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.201.156@tcp -r 1 -m PUT -l 6 [ 1459.618326] LNet: 1 local NIs in recovery (showing 1): 192.168.201.56@tcp [ 1460.384489] Lustre: DEBUG MARKER: Force oleg156-client.virtnet to discover 192.168.201.156@tcp (in background) [ 1460.798825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.156@tcp [ 1462.317281] Lustre: DEBUG MARKER: Force oleg156-server to discover 192.168.201.56@tcp [ 1463.910916] LNet: There was an unexpected network error while writing to 192.168.201.156: rc = -22 [ 1465.173758] Lustre: DEBUG MARKER: Wait for 70565 [ 1467.102084] Lustre: DEBUG MARKER: Finished wait on 70565 [ 1467.268880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1467.328901] LNet: There was an unexpected network error while writing to 192.168.201.156: rc = -22 [ 1467.349278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1467.727714] LNet: 71016:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1467.738078] LNetError: 71016:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1467.779584] LNet: Removed LNI 192.168.201.56@tcp [ 1468.140795] Key type .llcrypt unregistered [ 1468.143489] Key type ._llcrypt unregistered [ 1485.972486] Key type ._llcrypt registered [ 1485.974585] Key type .llcrypt registered [ 1487.224613] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 12:27:04 (1772472424) [ 1487.588259] Key type .llcrypt unregistered [ 1487.593682] Key type ._llcrypt unregistered [ 1493.812430] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1493.995910] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1494.008353] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1494.011985] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1496.744212] Key type ._llcrypt registered [ 1496.746369] Key type .llcrypt registered [ 1496.873078] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1496.891802] alg: No test for adler32 (adler32-zlib) [ 1497.844441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1497.937438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1497.960545] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 1498.856531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1498.876470] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1507.775362] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 12:27:24 (1772472444) [ 1507.925270] LNet: 73086:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1507.953726] LNetError: 73086:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1507.976681] LNet: Removed LNI 192.168.201.56@tcp [ 1507.999844] LNet: Removed LNI 10.1.2.3@tcp [ 1508.851489] Key type .llcrypt unregistered [ 1508.855092] Key type ._llcrypt unregistered [ 1515.052307] Key type ._llcrypt registered [ 1515.054514] Key type .llcrypt registered [ 1515.182903] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1515.200775] alg: No test for adler32 (adler32-zlib) [ 1516.139385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1516.186139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1516.207542] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 1516.825196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1516.845077] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1517.822427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.201.56@tcp [ 1518.880680] LNetError: Unexpected error -22 connecting to 192.168.201.56@tcp at host 192.168.201.56:988 [ 1524.413175] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 12:27:41 (1772472461) [ 1524.833822] LNet: 74225:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1524.849607] LNetError: 74225:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1524.859316] LNet: Removed LNI 192.168.201.56@tcp [ 1524.883740] LNet: Removed LNI 10.1.2.3@tcp [ 1525.596456] Key type .llcrypt unregistered [ 1525.598148] Key type ._llcrypt unregistered [ 1532.573280] Key type ._llcrypt registered [ 1532.574798] Key type .llcrypt registered [ 1532.894358] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1532.922863] alg: No test for adler32 (adler32-zlib) [ 1533.967782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1534.058645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1534.091820] LNet: Added LNI 192.168.201.56@tcp1 [8/256/0/180] [ 1534.993333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1535.028838] LNet: Added LNI 192.168.201.56@tcp2 [8/256/0/180] [ 1536.269406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.201.56@tcp1 --nid 192.168.201.56@tcp2 [ 1540.118412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.56@tcp1 192.168.201.56@tcp2 [ 1540.177915] LNetError: 75060:0:(lib-move.c:2264:lnet_handle_find_routed_path()) Unable to send message from 192.168.201.56@tcp1 to 192.168.201.56@tcp2 - Route table may be misconfigured [ 1540.219445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.56@tcp2 192.168.201.56@tcp1 [ 1548.721856] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 12:28:05 (1772472485) [ 1548.908619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1548.971627] LNet: 75634:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1548.988337] LNetError: 75634:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1548.999448] LNet: Removed LNI 192.168.201.56@tcp1 [ 1550.058405] LNet: Removed LNI 192.168.201.56@tcp2 [ 1550.120650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1550.181336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1550.205202] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 1550.953468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1550.969198] LNet: Added LNI 192.168.201.56@tcp1 [8/256/0/180] [ 1551.733829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.56@tcp [ 1551.813560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.56@tcp 192.168.201.56@tcp [ 1551.869303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.56@tcp1 192.168.201.56@tcp1 [ 1551.940578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.56@tcp -d 192.168.201.56@tcp -e network_timeout [ 1552.006921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.56@tcp -d 192.168.201.56@tcp1 -e network_timeout [ 1552.041541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.56@tcp1 -d 192.168.201.56@tcp -e network_timeout [ 1552.099825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.56@tcp1 -d 192.168.201.56@tcp1 -e network_timeout [ 1552.143459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.56@tcp [ 1552.159479] LNet: There was an unexpected network error while writing to 192.168.201.56: rc = -22 [ 1552.183943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1552.225909] LNet: 1 local NIs in recovery (showing 1): 192.168.201.56@tcp [ 1552.271310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1552.369581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1558.917488] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 12:28:15 (1772472495) [ 1559.176872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1559.259840] LNet: 76495:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1559.265718] LNetError: 76495:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1559.272663] LNetError: 76495:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1559.290055] LNet: Removed LNI 192.168.201.56@tcp [ 1560.359862] LNet: Removed LNI 192.168.201.56@tcp1 [ 1560.399391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1560.544280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1560.835956] LNet: 76533:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1561.224332] Key type .llcrypt unregistered [ 1561.229697] Key type ._llcrypt unregistered [ 1572.625345] Key type ._llcrypt registered [ 1572.627344] Key type .llcrypt registered [ 1573.686416] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 12:28:30 (1772472510) [ 1574.143183] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1574.156234] alg: No test for adler32 (adler32-zlib) [ 1575.097099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1576.669810] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1578.611464] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 12:28:35 (1772472515) [ 1578.713467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1578.742422] LNet: 77623:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1578.790458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1578.923820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1578.950065] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 1579.751820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1579.777770] LNet: Added LNI 192.168.201.56@tcp1 [8/256/0/180] [ 1580.711376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.56@tcp [ 1580.740958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.56@tcp1 [ 1580.772275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.56@tcp1 [ 1586.932326] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 12:28:43 (1772472523) [ 1589.126436] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg156-server [ 1590.844068] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 12:28:47 (1772472527) [ 1592.365686] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg156-server [ 1593.741540] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 12:28:50 (1772472530) [ 1595.495840] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg156-server [ 1596.873230] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 12:28:53 (1772472533) [ 1598.905134] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg156-server [ 1600.508223] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 12:28:57 (1772472537) [ 1602.446750] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg156-server [ 1604.232727] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 12:29:01 (1772472541) [ 1606.636945] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg156-server [ 1608.379720] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 12:29:05 (1772472545) [ 1610.143515] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg156-server [ 1611.706298] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 12:29:08 (1772472548) [ 1613.376607] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg156-server [ 1615.047198] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 12:29:12 (1772472552) [ 1616.986131] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg156-server [ 1618.826202] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 12:29:15 (1772472555) [ 1618.915957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1618.953229] LNet: 81501:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1618.960131] LNetError: 81501:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1618.973069] LNet: Removed LNI 192.168.201.56@tcp [ 1620.005898] LNet: Removed LNI 192.168.201.56@tcp1 [ 1620.090795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1620.164532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1620.204401] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 1621.127351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1621.182887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.56@tcp [ 1621.341420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1621.370091] LNet: 81655:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1621.384062] LNetError: 81655:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1621.387464] LNetError: 81655:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1621.419727] LNet: Removed LNI 192.168.201.56@tcp [ 1621.453403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1621.540707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1621.582486] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 1622.525800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.56@tcp [ 1622.679321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1622.701862] LNetError: 81800:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1623.719994] LNet: Removed LNI 192.168.201.56@tcp [ 1623.736250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1623.803454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1623.835071] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 1624.367371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1624.430754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.56@tcp [ 1624.584980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1624.615147] LNet: 81947:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1624.624158] LNet: 81947:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1625.704326] LNet: Removed LNI 192.168.201.56@tcp [ 1625.733024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1625.789851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1625.815269] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 1626.579864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.56@tcp [ 1626.750466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1626.772339] LNetError: 82092:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1626.777128] LNetError: 82092:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1627.818374] LNet: Removed LNI 192.168.201.56@tcp [ 1627.853935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1627.933265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1627.971922] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 1628.567636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1628.628503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.56@tcp [ 1628.797408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1628.812898] LNet: 82240:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1628.817228] LNet: 82240:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1629.862374] LNet: Removed LNI 192.168.201.56@tcp [ 1629.916723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1630.007784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1630.069965] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 1630.975942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.56@tcp [ 1631.248152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1631.272422] LNetError: 82385:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1631.279465] LNetError: 82385:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1632.304617] LNet: Removed LNI 192.168.201.56@tcp [ 1632.340328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1632.449147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1632.493022] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 1633.269660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1633.407372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.56@tcp [ 1634.916149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1636.007630] LNet: Removed LNI 192.168.201.56@tcp [ 1636.069410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1636.166454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1636.205737] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 1637.059483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.56@tcp [ 1637.370528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1637.402318] LNet: 82687:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1637.407282] LNet: 82687:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1638.437638] LNet: Removed LNI 192.168.201.56@tcp [ 1638.494864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1638.582244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1638.616848] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 1639.620022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1639.719278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.56@tcp [ 1641.184943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1641.203948] LNetError: 82843:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1641.210607] LNetError: 82843:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1642.282138] LNet: Removed LNI 192.168.201.56@tcp [ 1642.337529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1642.415909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1642.455694] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 1643.748974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.56@tcp [ 1644.104439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1645.162832] LNet: Removed LNI 192.168.201.56@tcp [ 1645.197917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1645.286402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1645.334637] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 1646.116309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1646.205310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.56@tcp [ 1646.482529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1647.526218] LNet: Removed LNI 192.168.201.56@tcp [ 1647.548477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1647.626526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1647.666385] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 1648.468476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.56@tcp [ 1648.765730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1649.831248] LNet: Removed LNI 192.168.201.56@tcp [ 1649.979237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1650.230748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1650.281993] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 1651.286144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1651.408740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1651.449618] LNet: Removed LNI 192.168.201.56@tcp [ 1651.506291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1651.609873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1651.648865] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 1652.591318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1661.321798] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 12:29:57 (1772472597) [ 1661.520922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1661.550395] LNet: 84117:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1661.556947] LNet: 84117:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 1661.566739] LNetError: 84117:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1661.577294] LNetError: 84117:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1661.613031] LNet: Removed LNI 192.168.201.56@tcp [ 1661.658264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1661.713718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1661.735994] LNet: Added LNI 192.168.201.56@tcp231 [8/256/0/180] [ 1663.365217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1663.426739] LNet: Removed LNI 192.168.201.56@tcp231 [ 1663.472493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1663.614897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1663.659592] LNet: Added LNI 192.168.201.56@tcp231 [8/256/0/180] [ 1663.668913] LNet: 84265:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1664.311592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1664.354841] LNet: Removed LNI 192.168.201.56@tcp231 [ 1664.391173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1664.441077] LNet: Added LNI 192.168.201.56@tcp231 [8/256/0/180] [ 1666.108884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1666.140829] LNet: Removed LNI 192.168.201.56@tcp231 [ 1666.170386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1666.249609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1666.319472] LNet: Added LNI 192.168.201.56@tcp231 [8/256/0/0] [ 1666.336404] LNet: 84418:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1666.768361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1666.806916] LNet: Removed LNI 192.168.201.56@tcp231 [ 1666.824791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1666.850598] LNet: Added LNI 192.168.201.56@tcp231 [8/256/0/0] [ 1667.870411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1667.903706] LNet: Removed LNI 192.168.201.56@tcp231 [ 1667.944245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1668.058084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1668.098664] LNet: Added LNI 192.168.201.56@tcp231 [8/256/0/60] [ 1668.103668] LNet: 84569:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1668.367368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1668.393471] LNet: Removed LNI 192.168.201.56@tcp231 [ 1668.412568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1668.442389] LNet: Added LNI 192.168.201.56@tcp231 [8/256/0/60] [ 1669.763723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1669.819950] LNet: Removed LNI 192.168.201.56@tcp231 [ 1669.878270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1670.003459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1670.056736] LNet: Added LNI 192.168.201.56@tcp231 [8/256/0/180] [ 1670.578056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1670.629893] LNet: Removed LNI 192.168.201.56@tcp231 [ 1670.658403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1670.692571] LNet: Added LNI 192.168.201.56@tcp231 [8/256/0/180] [ 1678.360427] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 12:30:15 (1772472615) [ 1678.527542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1678.582425] LNet: Removed LNI 192.168.201.56@tcp231 [ 1678.631741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1678.727375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1678.743226] LNet: Added LNI 192.168.201.56@tcp232 [8/256/0/180] [ 1679.359582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1679.404748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1679.436992] LNet: Removed LNI 192.168.201.56@tcp232 [ 1679.455734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1679.477605] LNet: Added LNI 192.168.201.56@tcp232 [8/256/0/180] [ 1680.085522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1680.121951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1680.147281] LNet: Removed LNI 192.168.201.56@tcp232 [ 1680.164339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1680.204593] LNet: Added LNI 192.168.201.56@tcp232 [8/256/0/180] [ 1680.922425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1680.957242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1680.988730] LNet: Removed LNI 192.168.201.56@tcp232 [ 1681.009585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1681.035268] LNet: Added LNI 192.168.201.56@tcp232 [8/256/0/180] [ 1681.683782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1681.734552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1681.773306] LNet: Removed LNI 192.168.201.56@tcp232 [ 1681.797605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1688.250765] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 12:30:24 (1772472624) [ 1689.799851] Key type .llcrypt unregistered [ 1689.803786] Key type ._llcrypt unregistered [ 1696.114696] Key type ._llcrypt registered [ 1696.118532] Key type .llcrypt registered [ 1696.392366] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1696.420537] alg: No test for adler32 (adler32-zlib) [ 1697.358974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1697.392362] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 1704.149449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.156@tcp [ 1712.039853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1712.074965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1712.105566] LNet: Added LNI 192.168.201.56@tcp1 [8/256/0/180] [ 1713.501685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.56@tcp -d 192.168.201.156@tcp -m GET -e local_interrupt -r 2 [ 1713.554737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.56@tcp -d 192.168.201.156@tcp1 -m GET -e local_interrupt -r 2 [ 1713.599539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.56@tcp -d 192.168.201.56@tcp -m GET -e local_interrupt -r 2 [ 1713.645406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.56@tcp -d 192.168.201.56@tcp1 -m GET -e local_interrupt -r 2 [ 1713.692630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.56@tcp1 -d 192.168.201.156@tcp -m GET -e local_interrupt -r 2 [ 1713.735675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.56@tcp1 -d 192.168.201.156@tcp1 -m GET -e local_interrupt -r 2 [ 1713.779649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.56@tcp1 -d 192.168.201.56@tcp -m GET -e local_interrupt -r 2 [ 1713.809391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.56@tcp1 -d 192.168.201.56@tcp1 -m GET -e local_interrupt -r 2 [ 1713.853852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.56@tcp 192.168.201.156@tcp [ 1713.894749] LNet: There was an unexpected network error while writing to 192.168.201.156: rc = -22 [ 1713.900844] LNet: 1 local NIs in recovery (showing 1): 192.168.201.56@tcp [ 1714.168962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1714.204490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1714.246044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1714.306788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1714.366314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1714.401342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1714.427147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1714.448750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1714.459177] LNet: 87222:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1714.474412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1714.509703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1714.563497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1714.639623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1714.673447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1714.723754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1715.352984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.56@tcp -d 192.168.201.156@tcp -m GET -e local_dropped -r 2 [ 1715.398425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.56@tcp -d 192.168.201.156@tcp1 -m GET -e local_dropped -r 2 [ 1715.442811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.56@tcp -d 192.168.201.56@tcp -m GET -e local_dropped -r 2 [ 1715.489673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.56@tcp -d 192.168.201.56@tcp1 -m GET -e local_dropped -r 2 [ 1715.528654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.56@tcp1 -d 192.168.201.156@tcp -m GET -e local_dropped -r 2 [ 1715.571274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.56@tcp1 -d 192.168.201.156@tcp1 -m GET -e local_dropped -r 2 [ 1715.654660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.56@tcp1 -d 192.168.201.56@tcp -m GET -e local_dropped -r 2 [ 1715.721454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.56@tcp1 -d 192.168.201.56@tcp1 -m GET -e local_dropped -r 2 [ 1715.784278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.56@tcp 192.168.201.156@tcp [ 1715.816169] LNet: There was an unexpected network error while writing to 192.168.201.156: rc = -22 [ 1716.077131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1716.103794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1716.127450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1716.170833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1716.217078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1716.288799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1716.338653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1716.404805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1716.444321] LNet: 87342:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1716.485851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1716.543888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1716.600100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1716.641646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1716.701708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1716.763053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1717.514747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.56@tcp -d 192.168.201.156@tcp -m GET -e local_aborted -r 2 [ 1717.569389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.56@tcp -d 192.168.201.156@tcp1 -m GET -e local_aborted -r 2 [ 1717.627648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.56@tcp -d 192.168.201.56@tcp -m GET -e local_aborted -r 2 [ 1717.682867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.56@tcp -d 192.168.201.56@tcp1 -m GET -e local_aborted -r 2 [ 1717.744061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.56@tcp1 -d 192.168.201.156@tcp -m GET -e local_aborted -r 2 [ 1717.799363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.56@tcp1 -d 192.168.201.156@tcp1 -m GET -e local_aborted -r 2 [ 1717.837888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.56@tcp1 -d 192.168.201.56@tcp -m GET -e local_aborted -r 2 [ 1717.870848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.56@tcp1 -d 192.168.201.56@tcp1 -m GET -e local_aborted -r 2 [ 1717.912637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.56@tcp 192.168.201.156@tcp [ 1717.934600] LNet: There was an unexpected network error while writing to 192.168.201.156: rc = -22 [ 1718.151671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1718.178368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1718.211072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1718.252183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1718.298184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1718.335327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1718.358344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1718.382205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1718.398034] LNet: 87463:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1718.429421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1718.503709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1718.549730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1718.577257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1718.611563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1718.677159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1719.291193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.56@tcp -d 192.168.201.156@tcp -m GET -e local_no_route -r 2 [ 1719.326585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.56@tcp -d 192.168.201.156@tcp1 -m GET -e local_no_route -r 2 [ 1719.366530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.56@tcp -d 192.168.201.56@tcp -m GET -e local_no_route -r 2 [ 1719.407907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.56@tcp -d 192.168.201.56@tcp1 -m GET -e local_no_route -r 2 [ 1719.449985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.56@tcp1 -d 192.168.201.156@tcp -m GET -e local_no_route -r 2 [ 1719.491149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.56@tcp1 -d 192.168.201.156@tcp1 -m GET -e local_no_route -r 2 [ 1719.536270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.56@tcp1 -d 192.168.201.56@tcp -m GET -e local_no_route -r 2 [ 1719.614448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.56@tcp1 -d 192.168.201.56@tcp1 -m GET -e local_no_route -r 2 [ 1719.682738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.56@tcp 192.168.201.156@tcp [ 1720.004622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1720.050207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1720.081749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1720.105359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1720.142390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1720.168654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1720.218312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1720.275604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1720.324052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1720.381639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1720.429861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1720.466248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1720.497720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1720.526088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1721.127423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.56@tcp -d 192.168.201.156@tcp -m GET -e local_timeout -r 2 [ 1721.161513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.56@tcp -d 192.168.201.156@tcp1 -m GET -e local_timeout -r 2 [ 1721.184475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.56@tcp -d 192.168.201.56@tcp -m GET -e local_timeout -r 2 [ 1721.217182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.56@tcp -d 192.168.201.56@tcp1 -m GET -e local_timeout -r 2 [ 1721.248596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.56@tcp1 -d 192.168.201.156@tcp -m GET -e local_timeout -r 2 [ 1721.276433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.56@tcp1 -d 192.168.201.156@tcp1 -m GET -e local_timeout -r 2 [ 1721.323545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.56@tcp1 -d 192.168.201.56@tcp -m GET -e local_timeout -r 2 [ 1721.371299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.56@tcp1 -d 192.168.201.56@tcp1 -m GET -e local_timeout -r 2 [ 1721.423425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.56@tcp 192.168.201.156@tcp [ 1721.453327] LNet: There was an unexpected network error while writing to 192.168.201.156: rc = -22 [ 1721.456683] LNet: Skipped 1 previous similar message [ 1721.793394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1721.828450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1721.868433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1721.911813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1721.948513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1721.999536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1722.062120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1722.095646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1722.112389] LNet: 87704:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1722.121513] LNet: 87704:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1722.148373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1722.190584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1722.236514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1722.277862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1722.313958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1722.352583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1722.976147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.56@tcp -d 192.168.201.156@tcp -m GET -e remote_dropped -r 2 [ 1723.006437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.56@tcp -d 192.168.201.156@tcp1 -m GET -e remote_dropped -r 2 [ 1723.034931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.56@tcp -d 192.168.201.56@tcp -m GET -e remote_dropped -r 2 [ 1723.067717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.56@tcp -d 192.168.201.56@tcp1 -m GET -e remote_dropped -r 2 [ 1723.102731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.56@tcp1 -d 192.168.201.156@tcp -m GET -e remote_dropped -r 2 [ 1723.129248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.56@tcp1 -d 192.168.201.156@tcp1 -m GET -e remote_dropped -r 2 [ 1723.163601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.56@tcp1 -d 192.168.201.56@tcp -m GET -e remote_dropped -r 2 [ 1723.203805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.56@tcp1 -d 192.168.201.56@tcp1 -m GET -e remote_dropped -r 2 [ 1723.237625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.56@tcp 192.168.201.156@tcp [ 1723.259421] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.156@tcp [ 1723.462986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1723.492334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1723.521320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1723.551676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1723.579912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1723.625525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1723.674215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1723.722095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1723.776809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1723.835955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1723.891743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1723.944859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1723.995230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1724.038642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1729.345064] LNet: 87899:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1729.354089] LNetError: 87899:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1729.359666] LNet: Removed LNI 192.168.201.56@tcp [ 1729.374848] LNet: Removed LNI 192.168.201.56@tcp1 [ 1729.742765] Key type .llcrypt unregistered [ 1729.744065] Key type ._llcrypt unregistered [ 1745.485328] Key type ._llcrypt registered [ 1745.487460] Key type .llcrypt registered [ 1746.683478] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 12:31:23 (1772472683) [ 1747.113662] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1747.148833] alg: No test for adler32 (adler32-zlib) [ 1748.075361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1748.142095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1748.160929] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 1748.815764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1748.841228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.56@tcp [ 1748.854366] LNetError: 88876:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.201.56@tcp to recovery [ 1748.904936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.201.56@tcp [ 1748.993960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1754.865772] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 12:31:31 (1772472691) [ 1755.287923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1755.314674] LNet: 89465:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1755.333441] LNetError: 89465:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1755.354059] LNet: Removed LNI 192.168.201.56@tcp [ 1755.388184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1755.448764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1755.479944] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 1756.328253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1756.354221] LNet: Added LNI 192.168.201.56@tcp1 [8/256/0/180] [ 1757.325618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.201.56@tcp [ 1757.354728] LNetError: 89732:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.201.56@tcp to recovery [ 1757.473025] LNet: 1 local NIs in recovery (showing 1): 192.168.201.56@tcp [ 1757.487527] LNet: There was an unexpected network error while writing to 192.168.201.56: rc = -22 [ 1758.684713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.201.56@tcp [ 1766.156484] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 12:31:43 (1772472703) [ 1767.595587] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg156-server [ 1769.376494] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 12:31:46 (1772472706) [ 1771.244882] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg156-server [ 1772.721890] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 12:31:49 (1772472709) [ 1774.208843] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg156-server [ 1775.695436] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 12:31:52 (1772472712) [ 1775.898968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1775.920807] LNet: 91353:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1775.932255] LNetError: 91353:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1776.991178] LNet: Removed LNI 192.168.201.56@tcp [ 1776.994232] LNetError: 91353:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1777.025650] LNet: Removed LNI 192.168.201.56@tcp1 [ 1777.054293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1777.119564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1777.168874] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 1777.867984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1777.957809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1778.014884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1778.127919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1778.195336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1778.245113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1785.019409] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 12:32:01 (1772472721) [ 1785.157362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1785.218060] LNet: 92075:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1785.230522] LNetError: 92075:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1785.268784] LNet: Removed LNI 192.168.201.56@tcp [ 1785.312417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1785.386491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1785.431598] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 1794.039599] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 12:32:10 (1772472730) [ 1794.269933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1794.317571] LNet: 92783:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1794.330312] LNetError: 92783:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1794.350890] LNet: Removed LNI 192.168.201.56@tcp [ 1794.388792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1794.442809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1794.464884] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 1804.343152] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 12:32:20 (1772472740) [ 1806.494293] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1808.151967] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 12:32:25 (1772472745) [ 1808.574706] LNet: 93819:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1808.583669] LNetError: 93819:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1808.597626] LNet: Removed LNI 192.168.201.56@tcp [ 1809.106847] Key type .llcrypt unregistered [ 1809.110049] Key type ._llcrypt unregistered [ 1816.718979] Key type ._llcrypt registered [ 1816.721606] Key type .llcrypt registered [ 1816.930946] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1816.948789] alg: No test for adler32 (adler32-zlib) [ 1817.859158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1817.882847] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 1825.478955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.156@tcp [ 1829.446638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.201.156@tcp [ 1829.698445] LNet: 94236:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1829.703657] LNetError: 94236:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1829.712985] LNet: Removed LNI 192.168.201.56@tcp [ 1830.029703] Key type .llcrypt unregistered [ 1830.031398] Key type ._llcrypt unregistered [ 1841.243646] Key type ._llcrypt registered [ 1841.246391] Key type .llcrypt registered [ 1842.329563] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1844.407886] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1846.037928] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 12:33:02 (1772472782) [ 1846.505449] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1846.524599] alg: No test for adler32 (adler32-zlib) [ 1847.487480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1847.602349] LNet: 95323:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1848.337487] Key type .llcrypt unregistered [ 1848.339189] Key type ._llcrypt unregistered [ 1854.044337] Key type ._llcrypt registered [ 1854.046833] Key type .llcrypt registered [ 1854.199394] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1854.213349] alg: No test for adler32 (adler32-zlib) [ 1855.096717] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 1855.099424] LNet: 95579:0:(config.c:1245:lnet_parse_routes()) Kernel parsing of LNet routes is deprecated. Consider converting to LNet YAML configuration. [ 1855.104735] LNet: 95579:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 1855.127065] LNetError: 95592:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) route to o2ib through 192.168.201.60@tcp has gone from up to down [ 1855.146432] LNetError: 95591:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.57@tcp is being used as a gateway but routing feature is not turned on [ 1855.659626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1858.783237] LNetError: 95591:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) route to o2ib through 192.168.201.61@tcp has gone from up to down [ 1858.800610] LNetError: 95591:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) Skipped 3 previous similar messages [ 1858.817401] LNet: 95637:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1858.840689] LNetError: 95637:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1858.873512] LNet: Removed LNI 192.168.201.56@tcp [ 1858.908441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1859.007333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1859.038142] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 1859.049255] LNet: 95645:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 1859.064210] LNet: 95645:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 1859.081110] LNetError: 95645:0:(peer.c:366:lnet_peer_ni_del_locked()) Peer NI 192.168.201.57@tcp is a gateway. Can not delete it [ 1859.089460] LNetError: 95645:0:(peer.c:2157:lnet_del_peer_ni()) 192.168.201.57@tcp is a router. Can not be deleted [ 1859.096939] LNet: 95645:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1859.106875] LNetError: 95642:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.59@tcp is being used as a gateway but routing feature is not turned on [ 1859.117552] LNetError: 95642:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 3 previous similar messages [ 1861.280765] LNetError: 95649:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) route to o2ib through 192.168.201.61@tcp has gone from up to down [ 1861.303552] LNetError: 95649:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) Skipped 3 previous similar messages [ 1866.328824] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 12:33:22 (1772472802) [ 1868.669528] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg156-server [ 1870.703430] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 12:33:27 (1772472807) [ 1872.546563] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg156-server [ 1874.116623] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 12:33:31 (1772472811) [ 1876.424730] Lustre: Lustre: Build Version: 2.17.50_196_gee1d501 [ 1878.415404] Key type lgssc registered [ 1880.106071] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1901.900996] Key type lgssc unregistered [ 1902.171772] LNet: 98069:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1902.177933] LNetError: 98069:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1903.206185] LNet: Removed LNI 192.168.201.56@tcp [ 1903.698814] Key type .llcrypt unregistered [ 1903.701634] Key type ._llcrypt unregistered [ 1919.392965] Key type ._llcrypt registered [ 1919.395491] Key type .llcrypt registered [ 1920.704908] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 12:34:17 (1772472857) [ 1921.181728] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1921.205262] alg: No test for adler32 (adler32-zlib) [ 1922.141453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1922.233680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1922.286373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1922.318623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1922.401311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1928.979775] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 12:34:25 (1772472865) [ 1929.106974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1929.139849] LNet: 99470:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1929.178492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1929.236253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1929.251409] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 1929.902202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1929.925969] LNet: 99619:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1929.933240] LNetError: 99619:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1929.945826] LNet: Removed LNI 192.168.201.56@tcp [ 1929.979258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1930.023968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1930.074156] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 1930.568526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1930.587480] LNetError: 99648:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1930.622241] LNet: Removed LNI 192.168.201.56@tcp [ 1930.640756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1930.691792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1930.725768] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 1938.094772] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 12:34:34 (1772472874) [ 1938.249297] LNet: 100231:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1938.262225] LNet: 100231:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1938.275645] LNetError: 100231:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1938.298401] LNet: Removed LNI 192.168.201.56@tcp [ 1939.302924] Key type .llcrypt unregistered [ 1939.305606] Key type ._llcrypt unregistered [ 1949.763246] Key type ._llcrypt registered [ 1949.776275] Key type .llcrypt registered [ 1950.102244] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1950.135711] alg: No test for adler32 (adler32-zlib) [ 1951.083382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1951.219067] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 1951.224604] LNet: *** cfs_fail_loc=e003, val=100*** [ 1952.726434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.56@tcp [ 1952.772503] LNet: *** cfs_fail_loc=e003, val=100*** [ 1952.785136] LNet: Skipped 2 previous similar messages [ 1952.828355] Ping reply received with mapping: [ 1952.834504] found metadata reply NI[192.168.201.56@tcp] [ 1952.844832] 38 c9 a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1952.850835] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1952.872850] 5a 5a 5a 5a 5a 5a 5a 5a [ 1953.256557] LNet: 100638:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1953.293472] LNetError: 100638:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1953.334466] LNet: Removed LNI 192.168.201.56@tcp [ 1954.508834] Key type .llcrypt unregistered [ 1954.534306] Key type ._llcrypt unregistered [ 1970.201841] Key type ._llcrypt registered [ 1970.203601] Key type .llcrypt registered [ 1971.679924] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 12:35:08 (1772472908) [ 1972.218494] Key type .llcrypt unregistered [ 1972.226661] Key type ._llcrypt unregistered [ 1978.808230] Key type ._llcrypt registered [ 1978.816154] Key type .llcrypt registered [ 1979.200626] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1979.233383] alg: No test for adler32 (adler32-zlib) [ 1980.321899] LNetError: 101685:0:(api-ni.c:2731:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1980.654371] Key type .llcrypt unregistered [ 1980.656080] Key type ._llcrypt unregistered [ 1985.677736] Key type ._llcrypt registered [ 1985.679645] Key type .llcrypt registered [ 1987.159188] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 12:35:23 (1772472923) [ 1987.611433] Key type .llcrypt unregistered [ 1987.612850] Key type ._llcrypt unregistered [ 1993.452907] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1993.457554] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1995.893970] Key type ._llcrypt registered [ 1995.896637] Key type .llcrypt registered [ 1996.072086] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1996.096967] alg: No test for adler32 (adler32-zlib) [ 1997.149121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1997.288268] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2007.112315] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 12:35:43 (1772472943) [ 2007.283215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2007.313912] LNet: 103462:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2007.326549] LNetError: 103462:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2007.363520] LNet: Removed LNI 10.1.2.3@tcp [ 2007.415810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2008.059078] Key type .llcrypt unregistered [ 2008.063061] Key type ._llcrypt unregistered [ 2014.674241] Key type ._llcrypt registered [ 2014.676078] Key type .llcrypt registered [ 2014.995907] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2015.047210] alg: No test for adler32 (adler32-zlib) [ 2016.119645] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 2016.249230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.56@tcp [ 2016.487445] LNet: 103753:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2016.501731] LNetError: 103753:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2016.525476] LNet: Removed LNI 192.168.201.56@tcp [ 2017.309859] Key type .llcrypt unregistered [ 2017.313827] Key type ._llcrypt unregistered [ 2030.275439] Key type ._llcrypt registered [ 2030.277040] Key type .llcrypt registered [ 2032.656206] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 12:36:08 (1772472968) [ 2033.329466] Key type .llcrypt unregistered [ 2033.332574] Key type ._llcrypt unregistered [ 2040.234677] Key type ._llcrypt registered [ 2040.236225] Key type .llcrypt registered [ 2040.502707] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2040.530451] alg: No test for adler32 (adler32-zlib) [ 2049.084987] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 12:36:25 (1772472985) [ 2049.232972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2049.296502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2049.359175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2049.383855] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 2049.942913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2049.969500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2049.993172] LNet: Removed LNI 192.168.201.56@tcp [ 2050.333266] LNet: 105498:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2050.699466] Key type .llcrypt unregistered [ 2050.702232] Key type ._llcrypt unregistered [ 2063.941480] Key type ._llcrypt registered [ 2063.945498] Key type .llcrypt registered [ 2065.593706] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 12:36:41 (1772473001) [ 2065.961531] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2065.969698] alg: No test for adler32 (adler32-zlib) [ 2066.851307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2066.890943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2066.917507] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 2067.562216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.56@tcp [ 2067.586321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.201.56@tcp [ 2067.611617] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2067.614778] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2067.619342] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2067, 0, 0 [ 2067.626964] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2067.629796] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2067.634892] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2067.638322] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2067, 0, 0 [ 2067.644576] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2067.648456] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2067.653180] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2067.662322] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2067, 0, 2117 [ 2067.671721] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2067.677387] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2067.686557] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2067.689619] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2067, 0, 2117 [ 2067.696971] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2073.654507] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 12:36:50 (1772473010) [ 2074.373854] LNet: 107038:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2074.379528] LNetError: 107038:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2075.431057] LNet: Removed LNI 192.168.201.56@tcp [ 2076.379532] Key type .llcrypt unregistered [ 2076.381365] Key type ._llcrypt unregistered [ 2084.294107] Key type ._llcrypt registered [ 2084.299626] Key type .llcrypt registered [ 2084.604609] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2084.653325] alg: No test for adler32 (adler32-zlib) [ 2085.568127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2085.590559] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 2095.138206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.156@tcp [ 2103.258144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2103.345104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2103.392694] LNet: Added LNI 192.168.201.56@tcp1 [8/256/0/180] [ 2104.211135] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2104.337373] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2104.349102] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2104.364054] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2106.835348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2106.850945] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2116.879535] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 12:37:33 (1772473053) [ 2117.198559] LNet: 108547:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2117.211994] LNetError: 108547:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2118.239767] LNet: Removed LNI 192.168.201.56@tcp [ 2118.244195] LNetError: 108547:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2118.264543] LNet: Removed LNI 192.168.201.56@tcp1 [ 2118.292472] LNet: Removed LNI 10.1.2.3@tcp99 [ 2119.207714] Key type .llcrypt unregistered [ 2119.211431] Key type ._llcrypt unregistered [ 2126.436756] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2126.456633] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2129.381980] Key type ._llcrypt registered [ 2129.386302] Key type .llcrypt registered [ 2129.641922] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2129.675736] alg: No test for adler32 (adler32-zlib) [ 2130.617424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2130.658154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2130.674576] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 2131.445646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2131.475268] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2134.290864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2134.444232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2134.507287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2134.566449] LNetError: 109292:0:(peer.c:2165:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2134.610454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2134.681377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.56@tcp [ 2135.319401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2135.478848] LNet: 109312:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2135.502069] LNetError: 109312:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2136.543162] LNet: Removed LNI 192.168.201.56@tcp [ 2136.544949] LNetError: 109312:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2136.588227] LNet: Removed LNI 10.1.2.3@tcp [ 2137.700901] Key type .llcrypt unregistered [ 2137.704883] Key type ._llcrypt unregistered [ 2153.541857] Key type ._llcrypt registered [ 2153.543748] Key type .llcrypt registered [ 2154.627732] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 12:38:11 (1772473091) [ 2154.954220] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2154.970245] alg: No test for adler32 (adler32-zlib) [ 2155.914770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2155.995658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2156.026896] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 2156.843967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.56@tcp [ 2156.898870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg156-client@tcp [ 2164.952584] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 12:38:20 (1772473100) [ 2165.038697] LNet: 110840:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2165.044780] LNetError: 110840:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2165.057966] LNet: Removed LNI 192.168.201.56@tcp [ 2165.846599] Key type .llcrypt unregistered [ 2165.849878] Key type ._llcrypt unregistered [ 2174.505785] Key type ._llcrypt registered [ 2174.516411] Key type .llcrypt registered [ 2174.705395] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2174.756282] alg: No test for adler32 (adler32-zlib) [ 2175.706890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2175.774466] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 2175.842284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2184.204853] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 12:38:40 (1772473120) [ 2186.553027] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2188.929113] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 12:38:45 (1772473125) [ 2190.929047] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2192.784903] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 12:38:49 (1772473129) [ 2193.270530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2193.325350] LNet: 112267:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2193.336067] LNetError: 112267:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2193.357399] LNet: Removed LNI 192.168.201.56@tcp [ 2193.388333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2193.517765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2193.639981] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 2195.476845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2195.516488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2204.476202] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 12:39:01 (1772473141) [ 2204.976214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2205.006402] LNet: 112977:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2205.015443] LNetError: 112977:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2205.038666] LNet: Removed LNI 192.168.201.56@tcp [ 2205.058793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2205.109832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2205.148146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2206.010307] Key type .llcrypt unregistered [ 2206.020888] Key type ._llcrypt unregistered [ 2220.207749] Key type ._llcrypt registered [ 2220.209481] Key type .llcrypt registered [ 2221.872457] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 12:39:18 (1772473158) [ 2222.209463] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2222.220531] alg: No test for adler32 (adler32-zlib) [ 2223.150379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2223.193353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2223.228530] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 2223.975121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2224.036375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.56@tcp [ 2230.963418] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 12:39:27 (1772473167) [ 2231.377351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2231.412249] LNet: 114533:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2231.422232] LNetError: 114533:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2231.451885] LNet: Removed LNI 192.168.201.56@tcp [ 2231.477251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2231.535143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2231.573694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2237.133486] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 12:39:34 (1772473174) [ 2237.279084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2237.320212] LNet: 115109:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2237.359461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2237.450628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2237.803534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2237.836763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2237.882806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2238.330460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2238.360930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2238.473286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2238.504342] LNet: 115146:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2238.523961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2238.634233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2239.062030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2239.086904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2239.150633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2239.436296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2239.472505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2246.182515] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 12:39:42 (1772473182) [ 2246.326715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2246.375728] LNet: 115740:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2246.407051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2246.466680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2246.492470] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 2247.187920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2247.208850] LNet: Added LNI 192.168.201.56@tcp2 [8/256/0/180] [ 2247.988405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2248.036320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.56@tcp [ 2248.218639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2248.244113] LNetError: 116026:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2248.250166] LNet: Removed LNI 192.168.201.56@tcp [ 2248.271064] LNet: Removed LNI 192.168.201.56@tcp2 [ 2248.299374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2248.373291] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 2248.381243] LNet: Added LNI 192.168.201.56@tcp2 [8/256/0/180] [ 2248.393450] LNet: 116032:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2248.973626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.56@tcp [ 2249.158565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2250.207557] LNet: Removed LNI 192.168.201.56@tcp [ 2250.210327] LNetError: 116069:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2250.216405] LNetError: 116069:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2250.232491] LNet: Removed LNI 192.168.201.56@tcp2 [ 2250.293691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2250.377644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2250.421799] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 2251.163741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2251.183059] LNet: Added LNI 192.168.201.56@tcp2 [8/256/0/180] [ 2252.118881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.56@tcp [ 2252.171549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2252.222894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.56@tcp [ 2252.421240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2252.438693] LNet: 116354:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2252.442926] LNet: 116354:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2252.446180] LNetError: 116354:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2253.471138] LNet: Removed LNI 192.168.201.56@tcp [ 2254.511977] LNet: Removed LNI 192.168.201.56@tcp2 [ 2254.538172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2254.616434] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 2254.622646] LNet: Added LNI 192.168.201.56@tcp2 [8/256/0/180] [ 2254.631104] LNet: 116360:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2254.922934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.56@tcp [ 2255.095213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2255.113397] LNet: Removed LNI 192.168.201.56@tcp [ 2256.165886] LNet: Removed LNI 192.168.201.56@tcp2 [ 2256.198433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2256.318420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2256.353590] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 2257.102628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2257.121343] LNet: Added LNI 192.168.201.56@tcp2 [8/256/0/180] [ 2257.711837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2257.759141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.56@tcp [ 2257.977228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2258.000586] LNetError: 116676:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2258.005969] LNetError: 116676:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2258.010373] LNet: Removed LNI 192.168.201.56@tcp [ 2258.019570] LNet: Removed LNI 192.168.201.56@tcp2 [ 2258.044699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2258.133337] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 2258.138653] LNet: Added LNI 192.168.201.56@tcp2 [8/256/0/180] [ 2258.143898] LNet: 116682:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2258.702693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.56@tcp [ 2258.922540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2260.002714] LNet: Removed LNI 192.168.201.56@tcp [ 2260.024965] LNet: Removed LNI 192.168.201.56@tcp2 [ 2260.080238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2260.150868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2260.187620] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 2260.856230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2260.872422] LNet: Added LNI 192.168.201.56@tcp2 [8/256/0/180] [ 2261.487372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.56@tcp [ 2261.518922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2261.565521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.56@tcp [ 2261.706728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2261.721697] LNet: 117004:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2261.725478] LNet: 117004:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2262.751162] LNet: Removed LNI 192.168.201.56@tcp [ 2263.783224] LNet: Removed LNI 192.168.201.56@tcp2 [ 2263.836595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2264.009645] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 2264.029604] LNet: Added LNI 192.168.201.56@tcp2 [8/256/0/180] [ 2264.039425] LNet: 117010:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2264.768748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.56@tcp [ 2265.014200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2265.029341] LNet: Removed LNI 192.168.201.56@tcp [ 2266.090217] LNet: Removed LNI 192.168.201.56@tcp2 [ 2266.292277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2266.540982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2266.602570] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 2268.060827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2268.102381] LNet: Added LNI 192.168.201.56@tcp2 [8/256/0/180] [ 2269.878192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2269.968333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2270.046195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.56@tcp [ 2270.745834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2270.810800] LNetError: 117344:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2270.825492] LNetError: 117344:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2270.840989] LNet: Removed LNI 192.168.201.56@tcp [ 2270.859226] LNet: Removed LNI 192.168.201.56@tcp2 [ 2270.957262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2271.137987] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 2271.147324] LNet: Added LNI 192.168.201.56@tcp2 [8/256/0/180] [ 2271.161884] LNet: 117350:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2272.429813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.56@tcp [ 2273.182933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2274.272228] LNet: Removed LNI 192.168.201.56@tcp [ 2274.314396] LNet: Removed LNI 192.168.201.56@tcp2 [ 2274.427798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2274.519575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2274.584149] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 2275.888993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2275.947504] LNet: Added LNI 192.168.201.56@tcp2 [8/256/0/180] [ 2277.493556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.56@tcp [ 2277.560735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2277.623424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2277.740454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.56@tcp [ 2278.134676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2278.157991] LNet: 117704:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2278.162731] LNet: 117704:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2279.199269] LNet: Removed LNI 192.168.201.56@tcp [ 2280.230767] LNet: Removed LNI 192.168.201.56@tcp2 [ 2280.295629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2280.387054] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 2280.391837] LNet: Added LNI 192.168.201.56@tcp2 [8/256/0/180] [ 2280.402868] LNet: 117710:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2281.218706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.56@tcp [ 2289.467817] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2291.610767] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 12:40:28 (1772473228) [ 2291.806258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2291.871393] LNetError: 118465:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2291.874091] LNetError: 118465:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2291.888779] LNet: Removed LNI 192.168.201.56@tcp [ 2292.971585] LNet: Removed LNI 192.168.201.56@tcp2 [ 2293.081313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2293.387776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2293.436917] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 2294.401353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2294.426529] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2295.690404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2295.717663] LNet: Removed LNI 192.168.201.56@tcp [ 2295.733280] LNet: Removed LNI 10.1.2.3@tcp [ 2295.767639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2295.869503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2295.891843] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 2295.895387] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2296.612853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2296.682808] LNet: Removed LNI 192.168.201.56@tcp [ 2296.732953] LNet: Removed LNI 10.1.2.3@tcp [ 2305.102185] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 12:40:41 (1772473241) [ 2305.248410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2305.328653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2305.502681] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2305.718184] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2305.748524] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2305.767846] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2308.382863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2308.417463] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 2309.349566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2309.388462] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2312.097872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.56@tcp [ 2312.145815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2312.230545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.201.56@tcp -m PUT -e local_timeout -r 1 [ 2312.307637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2312.374604] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2312.379088] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2312.566048] LNetError: 119409:0:(lib-move.c:4960:lnet_parse()) 192.168.201.56@tcp, src 192.168.201.56@tcp: Dropping PUT (error -108 looking up sender) [ 2313.567676] LNet: 119867:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2313.578957] LNet: 119867:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2314.657376] LNet: Removed LNI 192.168.201.56@tcp [ 2314.667739] LNet: Removed LNI 10.1.2.3@tcp [ 2315.167417] Key type .llcrypt unregistered [ 2315.176149] Key type ._llcrypt unregistered [ 2328.954332] Key type ._llcrypt registered [ 2328.960022] Key type .llcrypt registered [ 2330.608599] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 12:41:06 (1772473266) [ 2330.890563] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2330.908350] alg: No test for adler32 (adler32-zlib) [ 2331.784959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2332.253799] LNet: 120720:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2333.003682] Key type .llcrypt unregistered [ 2333.008506] Key type ._llcrypt unregistered [ 2338.701105] Key type ._llcrypt registered [ 2338.706887] Key type .llcrypt registered [ 2338.866374] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2338.876209] alg: No test for adler32 (adler32-zlib) [ 2339.852915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2339.903143] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 2345.827856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.156@tcp [ 2346.096527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2346.629970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.156@tcp [ 2346.869386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2346.900985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2346.935906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2346.965313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2347.007685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2347.044579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2347.077335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2347.103991] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.156@tcp [ 2347.122620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2347.134817] LNet: 121169:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2347.154366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2347.184941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2347.215903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2347.251621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2347.285848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2347.309254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2347.971599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2348.139746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2348.165098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2348.198391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2348.262222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2348.315300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2348.347617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2348.402047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2348.437597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2348.464236] LNet: 121286:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2348.487438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2348.532620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2348.600907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2348.643970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2348.686523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2348.738436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2349.366921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.156@tcp [ 2349.607410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2349.633140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2349.661372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2349.704510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2349.752946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2349.787350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2349.814705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2349.841929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2349.862557] LNet: 121404:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2349.882357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2349.916164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2349.946127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2349.981515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2350.014238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2350.048908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2350.623876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2350.954845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2351.004073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2351.040986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2351.070902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2351.123977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2351.168151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2351.230950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2351.272051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2351.336955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2351.405203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2351.441452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2351.488603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2351.551754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2351.597963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2351.915354] LNet: 121567:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2351.930268] LNetError: 121567:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2351.959329] LNet: Removed LNI 192.168.201.56@tcp [ 2352.439564] Key type .llcrypt unregistered [ 2352.441762] Key type ._llcrypt unregistered [ 2369.131450] Key type ._llcrypt registered [ 2369.134324] Key type .llcrypt registered [ 2370.181346] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 12:41:47 (1772473307) [ 2370.583564] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2370.592875] alg: No test for adler32 (adler32-zlib) [ 2371.513589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2371.843613] LNet: 122455:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2372.293799] Key type .llcrypt unregistered [ 2372.296118] Key type ._llcrypt unregistered [ 2377.801216] Key type ._llcrypt registered [ 2377.803297] Key type .llcrypt registered [ 2377.987813] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2378.002777] alg: No test for adler32 (adler32-zlib) [ 2379.029755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2379.081948] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 2386.571459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.156@tcp [ 2395.716979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2395.812233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2395.855705] LNet: Added LNI 192.168.201.56@tcp1 [8/256/0/180] [ 2397.098467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2397.796777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.156@tcp [ 2398.063610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2398.094782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2398.144218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2398.193401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2398.227717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2398.254993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2398.289598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2398.313222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2398.332397] LNet: 123112:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2398.345941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2398.373531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2398.400228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2398.434738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2398.461843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2398.493940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2399.210624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2399.565443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2399.585522] LNet: 2 peer NIs in recovery (showing 2): 192.168.201.156@tcp, 192.168.201.156@tcp1 [ 2399.608552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2399.652420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2399.686136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2399.733231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2399.788620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2399.849413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2399.891626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2399.924736] LNet: 123229:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2399.950089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2400.015657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2400.054803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2400.110463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2400.138802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2400.185209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2400.827365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.156@tcp [ 2401.008330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2401.032689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2401.068221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2401.108810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2401.153842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2401.182083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2401.206620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2401.235183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2401.258277] LNet: 123346:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2401.280864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2401.309749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2401.332420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2401.352302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2401.388990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2401.434934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2402.192693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2402.517631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2402.586959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2402.651599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2402.739997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2402.769522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2402.793562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2402.823533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2402.859663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2402.899468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2402.928380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2402.961664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2403.000099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2403.045799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2403.084733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2408.303630] LNet: 123544:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2408.309283] LNetError: 123544:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2409.375194] LNet: Removed LNI 192.168.201.56@tcp [ 2409.382027] LNetError: 123544:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2409.426890] LNet: Removed LNI 192.168.201.56@tcp1 [ 2410.141767] Key type .llcrypt unregistered [ 2410.143332] Key type ._llcrypt unregistered [ 2427.806756] Key type ._llcrypt registered [ 2427.817942] Key type .llcrypt registered [ 2429.358393] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 12:42:45 (1772473365) [ 2436.488524] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 12:42:53 (1772473373) [ 2443.618456] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 12:43:00 (1772473380) [ 2450.366244] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 12:43:07 (1772473387) [ 2457.466576] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 12:43:14 (1772473394) [ 2465.259224] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 12:43:21 (1772473401) [ 2472.016487] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 12:43:28 (1772473408) [ 2478.261504] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 12:43:35 (1772473415) [ 2485.366959] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 12:43:42 (1772473422) [ 2492.358313] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 12:43:49 (1772473429) [ 2498.772142] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 12:43:55 (1772473435) [ 2504.797900] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 12:44:01 (1772473441) [ 2510.862881] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 12:44:07 (1772473447) [ 2516.698629] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 12:44:13 (1772473453) [ 2523.055366] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 12:44:20 (1772473460) [ 2529.141444] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 12:44:26 (1772473466) [ 2535.736891] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 12:44:32 (1772473472) [ 2542.619799] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 12:44:39 (1772473479) [ 2543.680706] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2543.693303] alg: No test for adler32 (adler32-zlib) [ 2544.611854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2544.733554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2544.771298] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 2546.230866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2546.249604] LNet: 134252:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2546.254895] LNetError: 134252:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2546.275899] LNet: Removed LNI 192.168.201.56@tcp [ 2546.295941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2546.326924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2546.338079] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 2552.019649] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 12:44:49 (1772473489) [ 2558.314614] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 12:44:55 (1772473495) [ 2564.087825] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 12:45:01 (1772473501) [ 2569.046165] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 12:45:06 (1772473506) [ 2573.549681] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 12:45:10 (1772473510) [ 2580.001730] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 12:45:16 (1772473516) [ 2585.634555] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 12:45:22 (1772473522) [ 2590.375937] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 12:45:27 (1772473527) [ 2594.816526] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 12:45:32 (1772473532) [ 2599.249622] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 12:45:36 (1772473536) [ 2603.777042] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 12:45:41 (1772473541) [ 2607.602846] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 12:45:45 (1772473545) [ 2611.556260] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2374 sec ========== 12:45:48 (1772473548) [ 2611.702918] LNet: 141712:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2611.707645] LNetError: 141712:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2611.723222] LNet: Removed LNI 192.168.201.56@tcp [ 2612.081079] Key type .llcrypt unregistered [ 2612.083228] Key type ._llcrypt unregistered [ 2618.387219] Key type ._llcrypt registered [ 2618.389023] Key type .llcrypt registered [ 2618.620801] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2618.629730] alg: No test for adler32 (adler32-zlib) [ 2619.572995] Lustre: Lustre: Build Version: 2.17.50_196_gee1d501 [ 2619.706779] LNet: Added LNI 192.168.201.56@tcp [8/256/0/180] [ 2621.319269] Key type lgssc registered [ 2621.972582] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2657.202769] Lustre: Mounted lustre-client [ 2660.366443] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2670.478278] LustreError: 144164:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff8d8750b57800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 2670.540984] Lustre: Unmounted lustre-client [ 2682.478437] Key type lgssc unregistered [ 2682.698141] LNet: 144644:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2682.703837] LNetError: 144644:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2682.716216] LNet: Removed LNI 192.168.201.56@tcp [ 2683.179123] Key type .llcrypt unregistered [ 2683.180255] Key type ._llcrypt unregistered