[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 462454322 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524588K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003388] x2apic enabled [ 0.005004] Switched APIC routing to physical x2apic. [ 0.006016] kvm-guest: setup PV IPIs [ 0.009757] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.010000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.010021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.011015] pid_max: default: 32768 minimum: 301 [ 0.012143] LSM: Security Framework initializing [ 0.013066] Yama: becoming mindful. [ 0.014053] SELinux: Initializing. [ 0.015090] *** VALIDATE selinux *** [ 0.023767] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028043] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029121] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030119] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032072] *** VALIDATE tmpfs *** [ 0.033316] *** VALIDATE proc *** [ 0.035204] *** VALIDATE cgroup *** [ 0.036015] *** VALIDATE cgroup2 *** [ 0.037288] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038164] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040030] Spectre V2 : User space: Vulnerable [ 0.041008] Speculative Store Bypass: Vulnerable [ 0.044223] debug: unmapping init [mem 0xffffffff90e59000-0xffffffff90e60fff] [ 0.046181] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047739] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048027] ... version: 2 [ 0.049015] ... bit width: 48 [ 0.050016] ... generic registers: 4 [ 0.051013] ... value mask: 0000ffffffffffff [ 0.052015] ... max period: 00007fffffffffff [ 0.053014] ... fixed-purpose events: 3 [ 0.054015] ... event mask: 000000070000000f [ 0.055443] rcu: Hierarchical SRCU implementation. [ 0.057486] smp: Bringing up secondary CPUs ... [ 0.058598] x86: Booting SMP configuration: [ 0.059024] .... node #0, CPUs: #1 #2 #3 [ 0.063826] smp: Brought up 1 node, 4 CPUs [ 0.065017] smpboot: Max logical packages: 1 [ 0.066012] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.177451] node 0 deferred pages initialised in 109ms [ 0.180099] devtmpfs: initialized [ 0.181361] x86/mm: Memory block size: 128MB [ 0.186000] gcov: version magic: 0x41383552 [ 0.189350] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.195140] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.198472] pinctrl core: initialized pinctrl subsystem [ 0.201293] [ 0.202009] ************************************************************* [ 0.205020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.208019] ** ** [ 0.210020] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.213019] ** ** [ 0.216019] ** This means that this kernel is built to expose internal ** [ 0.219016] ** IOMMU data structures, which may compromise security on ** [ 0.221017] ** your system. ** [ 0.224020] ** ** [ 0.227018] ** If you see this message and you are not debugging the ** [ 0.229015] ** kernel, report this immediately to your vendor! ** [ 0.231017] ** ** [ 0.234019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.237017] ************************************************************* [ 0.239829] NET: Registered protocol family 16 [ 0.242567] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.247204] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.252100] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.258109] cpuidle: using governor menu [ 0.259971] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.260660] PCI: Using configuration type 1 for base access [ 0.263164] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.277069] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.278037] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.280028] cryptd: max_cpu_qlen set to 1000 [ 0.283355] ACPI: Added _OSI(Module Device) [ 0.285021] ACPI: Added _OSI(Processor Device) [ 0.287017] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.289014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.295132] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.301485] ACPI: Interpreter enabled [ 0.303083] ACPI: PM: (supports S0 S3 S4 S5) [ 0.305029] ACPI: Using IOAPIC for interrupt routing [ 0.306153] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.310519] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.321678] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.324047] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.326031] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.330115] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.336876] acpiphp: Slot [2] registered [ 0.338262] acpiphp: Slot [5] registered [ 0.340249] acpiphp: Slot [6] registered [ 0.342191] acpiphp: Slot [3] registered [ 0.343165] acpiphp: Slot [4] registered [ 0.345089] acpiphp: Slot [7] registered [ 0.346091] acpiphp: Slot [8] registered [ 0.347102] acpiphp: Slot [9] registered [ 0.349113] acpiphp: Slot [10] registered [ 0.350150] acpiphp: Slot [11] registered [ 0.352147] acpiphp: Slot [12] registered [ 0.354185] acpiphp: Slot [13] registered [ 0.355137] acpiphp: Slot [14] registered [ 0.356118] acpiphp: Slot [15] registered [ 0.358345] acpiphp: Slot [16] registered [ 0.360133] acpiphp: Slot [17] registered [ 0.361151] acpiphp: Slot [18] registered [ 0.363236] acpiphp: Slot [19] registered [ 0.365145] acpiphp: Slot [20] registered [ 0.367165] acpiphp: Slot [21] registered [ 0.369205] acpiphp: Slot [22] registered [ 0.371174] acpiphp: Slot [23] registered [ 0.372148] acpiphp: Slot [24] registered [ 0.374133] acpiphp: Slot [25] registered [ 0.376213] acpiphp: Slot [26] registered [ 0.379176] acpiphp: Slot [27] registered [ 0.380137] acpiphp: Slot [28] registered [ 0.382215] acpiphp: Slot [29] registered [ 0.383173] acpiphp: Slot [30] registered [ 0.385160] acpiphp: Slot [31] registered [ 0.387099] PCI host bridge to bus 0000:00 [ 0.389042] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.392032] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.394027] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.397034] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.399032] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.403055] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.405207] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.408135] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.412371] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.419416] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.423729] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.427027] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.430023] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.433033] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.436770] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.440883] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.443060] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.446994] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.451992] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.462021] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.466042] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.472154] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.484016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.491017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.517016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.532060] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.543020] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.548020] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.564020] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.581142] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.591979] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.595722] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.598641] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.602420] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.621075] iommu: Default domain type: Passthrough [ 0.622545] SCSI subsystem initialized [ 0.627295] ACPI: bus type USB registered [ 0.630223] usbcore: registered new interface driver usbfs [ 0.633129] usbcore: registered new interface driver hub [ 0.635161] usbcore: registered new device driver usb [ 0.639239] pps_core: LinuxPPS API ver. 1 registered [ 0.641013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.644071] PTP clock support registered [ 0.646180] EDAC MC: Ver: 3.0.0 [ 0.649289] PCI: Using ACPI for IRQ routing [ 0.652074] NetLabel: Initializing [ 0.654014] NetLabel: domain hash size = 128 [ 0.655016] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.657108] NetLabel: unlabeled traffic allowed by default [ 0.660166] vgaarb: loaded [ 0.661328] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.662026] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.666350] clocksource: Switched to clocksource kvm-clock [ 0.817727] VFS: Disk quotas dquot_6.6.0 [ 0.819639] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.822624] *** VALIDATE ramfs *** [ 0.824808] *** VALIDATE hugetlbfs *** [ 0.826743] pnp: PnP ACPI init [ 0.830717] pnp: PnP ACPI: found 6 devices [ 0.851810] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.857593] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.862419] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.865849] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.869803] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.872829] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.876326] NET: Registered protocol family 2 [ 0.879608] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.886232] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.890178] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.896491] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.900320] TCP: Hash tables configured (established 65536 bind 65536) [ 0.903989] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.907568] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.910754] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.914587] NET: Registered protocol family 1 [ 0.918058] RPC: Registered named UNIX socket transport module. [ 0.925172] RPC: Registered udp transport module. [ 0.927753] RPC: Registered tcp transport module. [ 0.930723] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.939062] NET: Registered protocol family 44 [ 0.945727] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.950233] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.954705] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.962111] PCI: CLS 0 bytes, default 64 [ 0.966955] Unpacking initramfs... [ 4.831953] debug: unmapping init [mem 0xffff99363cc64000-0xffff99363ffcffff] [ 4.848986] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 4.851215] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 4.868349] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 7.205591] Initialise system trusted keyrings [ 7.216118] Key type blacklist registered [ 7.225765] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 7.248745] zbud: loaded [ 7.263668] *** VALIDATE nfs *** [ 7.265430] *** VALIDATE nfs4 *** [ 7.270281] pstore: using deflate compression [ 7.286356] Platform Keyring initialized [ 7.828625] NET: Registered protocol family 38 [ 7.833475] Key type asymmetric registered [ 7.839470] Asymmetric key parser 'x509' registered [ 7.844448] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 7.855164] io scheduler mq-deadline registered [ 7.860433] io scheduler kyber registered [ 7.871687] io scheduler bfq registered [ 7.878539] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 7.887983] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 7.897626] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 7.909160] ACPI: Power Button [PWRF] [ 7.928451] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 7.962740] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 7.988830] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 8.039135] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 8.083576] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 8.092893] Non-volatile memory driver v1.3 [ 8.100293] Linux agpgart interface v0.103 [ 8.153827] virtio_blk virtio1: [vda] 146640 512-byte logical blocks (75.1 MB/71.6 MiB) [ 8.161018] vda: detected capacity change from 0 to 75079680 [ 8.239299] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 8.254176] vdb: detected capacity change from 0 to 1073741824 [ 8.293853] libphy: Fixed MDIO Bus: probed [ 8.319215] usbcore: registered new interface driver usbserial_generic [ 8.327282] usbserial: USB Serial support registered for generic [ 8.344183] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 8.365148] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 8.367213] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 8.375788] mousedev: PS/2 mouse device common for all mice [ 8.386714] rtc_cmos 00:05: RTC can wake from S4 [ 8.399670] rtc_cmos 00:05: registered as rtc0 [ 8.410223] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 8.422084] intel_pstate: CPU model not supported [ 8.424474] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 8.437223] hid: raw HID events driver (C) Jiri Kosina [ 8.448579] usbcore: registered new interface driver usbhid [ 8.452075] usbhid: USB HID core driver [ 8.457584] drop_monitor: Initializing network drop monitor service [ 8.461868] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 8.465178] Initializing XFRM netlink socket [ 8.486203] NET: Registered protocol family 10 [ 8.489725] Segment Routing with IPv6 [ 8.491268] NET: Registered protocol family 17 [ 8.494338] mpls_gso: MPLS GSO support [ 8.499963] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 8.504984] RAS: Correctable Errors collector initialized. [ 8.507621] AVX version of gcm_enc/dec engaged. [ 8.509821] AES CTR mode by8 optimization enabled [ 8.671287] sched_clock: Marking stable (8671230358, 0)->(9636453126, -965222768) [ 8.686993] registered taskstats version 1 [ 8.692711] Loading compiled-in X.509 certificates [ 8.697011] zswap: loaded using pool lzo/zbud [ 8.774044] Key type big_key registered [ 8.801386] Key type encrypted registered [ 8.805622] ima: No TPM chip found, activating TPM-bypass! [ 8.813677] ima: Allocated hash algorithm: sha1 [ 8.819116] ima: No architecture policies found [ 8.831062] evm: Initialising EVM extended attributes: [ 8.836151] evm: security.selinux [ 8.839648] evm: security.ima [ 8.845703] evm: security.capability [ 8.853495] evm: HMAC attrs: 0x1 [ 8.864332] rtc_cmos 00:05: setting system clock to 2026-08-31 14:54:12 UTC (1788188052) [ 8.883792] debug: unmapping init [mem 0xffffffff91e03000-0xffffffff91ffffff] [ 8.899714] debug: unmapping init [mem 0xffffffff90b82000-0xffffffff90e58fff] [ 8.916153] Write protecting the kernel read-only data: 28672k [ 8.922091] debug: unmapping init [mem 0xffffffff8f203000-0xffffffff8f3fffff] [ 8.933319] debug: unmapping init [mem 0xffffffff8fb14000-0xffffffff8fbfffff] [ 9.062971] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 9.091466] systemd[1]: Detected virtualization kvm. [ 9.095632] systemd[1]: Detected architecture x86-64. [ 9.103637] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 9.213684] systemd[1]: No hostname configured. [ 9.215437] systemd[1]: Set hostname to . [ 9.217000] random: systemd: uninitialized urandom read (16 bytes read) [ 9.226556] systemd[1]: Initializing machine ID from random generator. [ 9.386573] random: ln: uninitialized urandom read (6 bytes read) [ 9.677275] random: systemd: uninitialized urandom read (16 bytes read) [ 9.684260] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 9.704841] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 9.732560] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Paths. Starting Journal Service... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Swap. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Slices. [ OK ] Reached target Timers. [ OK ] Listening on udev Control Socket. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... Starting Setup Virtual Console... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 11.423298] device-mapper: uevent: version 1.0.3 [ 11.427538] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook...[ 13.246453] random: fast init done [ 13.267511] virtio_net virtio0 ens2: renamed from eth0 [ 13.508270] scsi host0: ata_piix [ 13.533378] scsi host1: ata_piix [ 13.537707] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 13.543372] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 18.387177] random: crng init done [ 18.394699] random: 7 urandom warning(s) missed due to ratelimiting [ 19.753235] dracut-initqueue[586]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 21.195660] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 23.844762] printk: systemd: 26 output lines suppressed due to ratelimiting [ 24.688690] SELinux: Disabled at runtime. [ 24.811569] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 24.825129] systemd[1]: Detected virtualization kvm. [ 24.829094] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 26.275989] systemd[1]: initrd-switch-root.service: Succeeded. [ 26.282214] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 26.290547] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 26.298288] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 26.308710] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 26.334223] systemd[1]: Starting Journal Service... Starting Journal Service... [ 26.339839] systemd[1]: Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on RPCbind Server Activation Socket. Mounting Huge Pages File System... [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice User and Session Slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting Kernel Debug File System... [ OK ] Created slice system-getty.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-serial\x2dgetty.slice. Starting udev Coldplug all Devices... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Remount Root and Kernel File Systems... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. Mounting POSIX Message Queue File System... [ OK ] Reached target Slices. Activating swap /dev/disk/by-label/SWAP... Starting Apply Kernel Variables... [ 26.981279] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 27.741868] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 28.474934] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 28.491539] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 28.684248] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 28.764872] EDAC sbridge: Ver: 1.1.2 [ 32.304346] Key type dns_resolver registered [ 32.939680] NFS: Registering the id_resolver key type [ 32.944077] Key type id_resolver registered [ 32.946454] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg351-client login: [ 98.185303] libcfs: loading out-of-tree module taints kernel. [ 98.373319] Key type ._llcrypt registered [ 98.377896] Key type .llcrypt registered [ 98.958967] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 98.976431] alg: No test for adler32 (adler32-zlib) [ 100.626442] Lustre: Lustre: Build Version: 2.17.57_85_gb71fc30 [ 101.691977] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 103.539972] Key type lgssc registered [ 105.029110] Lustre: Echo OBD driver; http://www.lustre.org/ [ 235.637817] Lustre: Mounted lustre-client - version 2.17.57_85_gb71fc30 [ 240.304364] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 253.694541] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing check_logdir /tmp/testlogs/ [ 258.618470] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing yml_node [ 261.090186] Lustre: lustre-OST0000-osc-ffff9936885e6800: disconnect after 23s idle [ 262.430938] Lustre: DEBUG MARKER: Client: 2.17.57.85 [ 264.953871] Lustre: DEBUG MARKER: MDS: 2.17.57.85 [ 267.349970] Lustre: DEBUG MARKER: OSS: 2.17.57.85 [ 268.852173] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Mon Aug 31 10:58:31 EDT 2026 [ 281.856045] Lustre: Unmounted lustre-client [ 302.324594] Key type lgssc unregistered [ 302.580089] LNet: 8411:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 302.588192] LNetError: 8411:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 302.608379] LNet: Removed LNI 192.168.203.51@tcp [ 303.547616] Key type .llcrypt unregistered [ 303.549714] Key type ._llcrypt unregistered [ 327.827763] Key type ._llcrypt registered [ 327.839119] Key type .llcrypt registered [ 328.079719] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 328.125110] alg: No test for adler32 (adler32-zlib) [ 328.971465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 329.013911] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 329.031995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 332.752336] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 332.835960] LNet: 9283:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 332.851947] LNetError: 9283:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 332.881102] LNet: Removed LNI 192.168.203.51@tcp [ 333.601743] Key type .llcrypt unregistered [ 333.609122] Key type ._llcrypt unregistered [ 340.270085] Key type ._llcrypt registered [ 340.273373] Key type .llcrypt registered [ 341.972860] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 10:59:43 (1788188383) [ 342.508201] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 342.526751] alg: No test for adler32 (adler32-zlib) [ 343.514600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 343.663421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 343.697099] LNet: 9700:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 343.725737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 343.821942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 343.883768] LNet: 9707:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 353.144894] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 10:59:54 (1788188394) [ 353.347230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 353.384063] LNet: 10287:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 353.457795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 353.576924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 360.504069] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 11:00:02 (1788188402) [ 360.572373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 360.606205] LNet: 10868:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 360.656652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 360.781513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 366.118381] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 11:00:08 (1788188408) [ 366.288593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 366.307374] LNet: 11459:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 366.321971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 366.389738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 371.706309] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 11:00:14 (1788188414) [ 371.829611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 371.868239] LNet: 12043:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 371.884335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 371.937595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 372.248927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 377.277862] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 11:00:19 (1788188419) [ 377.562168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 377.602591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 377.666338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 378.108663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 378.506686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 384.477138] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 11:00:26 (1788188426) [ 384.554041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 384.575292] LNet: 13221:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 384.578328] LNet: 13221:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 384.596891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 384.693145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 390.826144] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 11:00:33 (1788188433) [ 390.894216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 390.964065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 391.085882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 391.135513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 391.572983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 391.618578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 392.176736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 392.229151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 392.520205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 392.578310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 393.157243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 393.197661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 393.666263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 393.703979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 394.146396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 394.179469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 394.666887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 394.705910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 395.135804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 395.178515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 395.552710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 395.590887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 395.939224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 395.982983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 396.259941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 396.288517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 396.659454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 396.693461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 402.608540] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 11:00:44 (1788188444) [ 402.761256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 402.790343] LNet: 14495:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 402.803745] LNet: 14495:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 402.836966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 402.948246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 402.997593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 408.838613] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 11:00:51 (1788188451) [ 408.974254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 409.013117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 409.111741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 409.180728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 415.329561] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 11:00:57 (1788188457) [ 415.577302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 415.609970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 415.671973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 415.720143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 422.369579] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 11:01:04 (1788188464) [ 422.553464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 422.647596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 422.748960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 422.776218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 429.213851] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 11:01:11 (1788188471) [ 429.299054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 429.328956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 429.427233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 429.464453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 430.858026] hrtimer: interrupt took 2125131 ns [ 435.663559] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 11:01:18 (1788188478) [ 435.802756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 435.853514] LNet: 17438:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 435.859926] LNet: 17438:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 435.883853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 435.958918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 435.991262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 442.653664] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 11:01:24 (1788188484) [ 442.847649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 442.904766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 442.989870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 443.506640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 449.926413] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 11:01:32 (1788188492) [ 450.104497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 450.139948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 450.258511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 450.453441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 456.781576] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 11:01:39 (1788188499) [ 456.910169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 456.972845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 457.140567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 457.374348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 462.726588] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 11:01:45 (1788188505) [ 462.841649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 462.901595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 463.305291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 463.607843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 470.454153] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 11:01:52 (1788188512) [ 470.592529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 470.650561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 470.857554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 470.948672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 477.590085] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 11:02:00 (1788188520) [ 477.918940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 477.965965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 478.068480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 478.111987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 484.443544] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 11:02:06 (1788188526) [ 484.612105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 484.689294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 484.774616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 491.196969] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 11:02:13 (1788188533) [ 491.353042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 491.407417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 491.482244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 497.239912] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 11:02:19 (1788188539) [ 497.341310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 497.408241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 497.518281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 497.933906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 503.725957] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 11:02:26 (1788188546) [ 503.985291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 504.014635] LNet: 23464:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 504.023502] LNet: 23464:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 504.060663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 504.175307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 504.237105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 511.109351] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 11:02:33 (1788188553) [ 511.273382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 511.349898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 511.475804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 511.509235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 518.060965] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 11:02:40 (1788188560) [ 518.270215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 518.309738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 518.372774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 518.417406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 524.844500] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 11:02:46 (1788188566) [ 524.918922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 524.975455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 525.101927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 525.130935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 531.136234] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 11:02:53 (1788188573) [ 531.371809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 531.399577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 531.441630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 531.510900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 531.530789] LNetError: 25821:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 531.559513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 531.581546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 537.013847] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 11:02:59 (1788188579) [ 537.225644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 537.285847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 537.356680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.0WQKGQN7oB [ 537.453767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.0WQKGQN7oB [ 537.524363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.0WQKGQN7oB [ 543.267556] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 11:03:05 (1788188585) [ 543.409782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 543.469108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 543.558693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 543.617488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 543.663214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 548.765641] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 11:03:11 (1788188591) [ 548.822968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 548.880826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 548.930662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 548.946464] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 550.565853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.51@tcp [ 551.915620] Lustre: Lustre: Build Version: 2.17.57_85_gb71fc30 [ 553.975483] Key type lgssc registered [ 555.691712] Lustre: Echo OBD driver; http://www.lustre.org/ [ 559.824324] LNetError: 27582:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.203.51@tcp99 has no available nets [ 563.161972] Key type lgssc unregistered [ 563.593260] LNetError: 28688:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 564.647459] LNet: Removed LNI 192.168.203.51@tcp [ 565.562901] Key type .llcrypt unregistered [ 565.565127] Key type ._llcrypt unregistered [ 572.849046] Key type ._llcrypt registered [ 572.852746] Key type .llcrypt registered [ 574.150901] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 11:03:36 (1788188616) [ 574.659265] Key type .llcrypt unregistered [ 574.661831] Key type ._llcrypt unregistered [ 580.387374] Key type ._llcrypt registered [ 580.391488] Key type .llcrypt registered [ 580.612425] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 580.640483] alg: No test for adler32 (adler32-zlib) [ 581.499372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 581.620874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 581.644830] LNet: 29598:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 581.650436] LNet: 29598:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 581.656472] LNet: 29598:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 581.667445] LNet: Message forwarding will be enabled in 2s [ 584.735160] LNet: Message forwarding enabled [ 585.168296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 585.201849] LNet: Message forwarding will stop in 2s to 6s [ 588.837574] LNet: Message forwarding disabled [ 589.725650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 589.746996] LNet: 29696:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 589.770623] LNet: 29696:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 589.780072] LNet: 29696:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 589.805215] LNet: Message forwarding will be enabled in 2s [ 590.028596] LNet: 29709:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 590.725492] Key type .llcrypt unregistered [ 590.727872] Key type ._llcrypt unregistered [ 601.538108] Key type ._llcrypt registered [ 601.543149] Key type .llcrypt registered [ 602.702853] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 11:04:05 (1788188645) [ 603.318568] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 603.337865] alg: No test for adler32 (adler32-zlib) [ 604.208297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 604.281235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 610.188218] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 11:04:12 (1788188652) [ 610.405484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 610.429407] LNet: 31109:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 610.456920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 610.533610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 610.569744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 610.603538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 610.636472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 610.674518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 610.711100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 610.747977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 610.778904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 610.843830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 610.916323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 610.947788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 611.007354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 611.040083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 611.073785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 611.113567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 611.170560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 611.215135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 611.266149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 611.315310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 611.338961] LNetError: 31152:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 617.116796] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 11:04:19 (1788188659) [ 617.253575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 617.289750] LNet: 31718:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 617.331101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 617.438429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 623.219550] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 11:04:25 (1788188665) [ 624.875582] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg351-server [ 626.360646] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 11:04:28 (1788188668) [ 626.509477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 626.542185] LNet: 32653:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 626.567067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 626.624786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 626.654796] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 627.487833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.203.[52-56]@tcp [ 627.514282] LNet: 32786:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 627.595824] LNetError: 32657:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.53@tcp is being used as a gateway but routing feature is not turned on [ 627.632987] LNetError: 32657:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.53@tcp has gone from up to down [ 634.670944] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 11:04:36 (1788188676) [ 636.607307] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg351-server [ 638.166718] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 11:04:40 (1788188680) [ 638.315756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 638.336888] LNet: 33719:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 638.342184] LNetError: 33719:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 638.357366] LNet: Removed LNI 192.168.203.51@tcp [ 638.407537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 638.458664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 638.478315] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 639.083265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.203.[52-57/2]@tcp [ 639.102967] LNet: 33840:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 639.108332] LNet: 33840:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 639.133528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.203.[52-57/2]@tcp [ 639.133913] LNetError: 33727:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.52@tcp is being used as a gateway but routing feature is not turned on [ 639.170035] LNetError: 33727:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 639.177283] LNetError: 33727:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp103 through 192.168.203.52@tcp has gone from up to down [ 639.188128] LNetError: 33727:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 645.305584] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 11:04:47 (1788188687) [ 645.417966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 645.444081] LNet: 34426:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 645.449746] LNetError: 34426:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 645.493918] LNet: Removed LNI 192.168.203.51@tcp [ 645.555513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 645.646145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 645.687923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 645.719791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 645.792929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 645.834976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 645.868045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 645.918932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 646.014171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 646.077296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 646.171567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 646.258210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 646.314242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 646.384919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 646.483443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 646.520447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 646.596307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 646.669855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 646.711845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 646.769405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 646.873647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 646.907974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 646.985650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 647.071572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 647.113913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 647.187396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 647.263565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 647.301892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 647.350467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 647.435986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 647.467422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 647.524539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 647.592696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 647.627941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 647.696447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 653.304651] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 11:04:55 (1788188695) [ 654.668710] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg351-server [ 656.287828] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 11:04:58 (1788188698) [ 657.749225] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg351-server [ 659.348982] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 11:05:01 (1788188701) [ 659.457889] LNet: 35825:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 659.466835] LNet: 35825:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 659.963221] Key type .llcrypt unregistered [ 659.966119] Key type ._llcrypt unregistered [ 666.782237] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 666.863732] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 666.869847] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 666.879899] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 669.574754] Key type ._llcrypt registered [ 669.581643] Key type .llcrypt registered [ 669.778864] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 669.793452] alg: No test for adler32 (adler32-zlib) [ 670.706923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 670.778106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 670.810593] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 671.412438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 671.437739] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 673.482916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 673.500485] LNet: Removed LNI 10.1.2.3@tcp [ 673.661771] LNet: 36551:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 673.673866] LNetError: 36551:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 673.693751] LNet: Removed LNI 192.168.203.51@tcp [ 674.317640] Key type .llcrypt unregistered [ 674.320432] Key type ._llcrypt unregistered [ 684.987785] Key type ._llcrypt registered [ 684.993707] Key type .llcrypt registered [ 686.400385] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 11:05:28 (1788188728) [ 686.838456] Key type .llcrypt unregistered [ 686.841011] Key type ._llcrypt unregistered [ 693.396338] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 693.573220] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 693.583977] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 693.597244] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 696.303283] Key type ._llcrypt registered [ 696.307183] Key type .llcrypt registered [ 696.522025] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 696.553466] alg: No test for adler32 (adler32-zlib) [ 697.517407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 697.601747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 697.659716] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 698.430885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 698.456069] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 700.960611] LNet: 38042:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 700.971256] LNetError: 38042:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 700.979562] LNet: Removed LNI 10.1.2.3@tcp [ 700.988868] LNet: Removed LNI 192.168.203.51@tcp [ 701.646687] Key type .llcrypt unregistered [ 701.648932] Key type ._llcrypt unregistered [ 713.047375] Key type ._llcrypt registered [ 713.053071] Key type .llcrypt registered [ 714.295387] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 11:05:56 (1788188756) [ 714.824275] Key type .llcrypt unregistered [ 714.827621] Key type ._llcrypt unregistered [ 720.760155] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 720.862380] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 720.865905] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 720.869466] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 723.652102] Key type ._llcrypt registered [ 723.654294] Key type .llcrypt registered [ 723.839585] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 723.854368] alg: No test for adler32 (adler32-zlib) [ 724.814476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 724.896533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 724.934257] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 727.114716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 727.140172] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 728.953150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 728.986184] LNet: Removed LNI 10.1.2.3@tcp [ 728.999125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 729.030173] LNet: Removed LNI 10.1.2.31@tcp [ 729.049488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 729.079539] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 729.801074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 729.834158] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 730.685159] LNet: 39955:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 730.693719] LNetError: 39955:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 730.700041] LNet: Removed LNI 10.1.2.31@tcp [ 730.712235] LNet: Removed LNI 10.1.2.3@tcp [ 731.569487] Key type .llcrypt unregistered [ 731.577469] Key type ._llcrypt unregistered [ 742.784782] Key type ._llcrypt registered [ 742.787960] Key type .llcrypt registered [ 743.953548] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 11:06:26 (1788188786) [ 744.347595] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 744.377573] alg: No test for adler32 (adler32-zlib) [ 745.271273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 745.322564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 745.340894] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 746.061581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 746.095731] LNet: 40905:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 746.100947] LNetError: 40905:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 746.122395] LNet: Removed LNI 192.168.203.51@tcp [ 746.143540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 746.218338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.203.51@tcp [ 746.237280] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 746.928590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.203.51@tcp [ 746.963533] LNet: Removed LNI 192.168.203.51@tcp [ 747.019715] LNet: 41041:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 747.636697] Key type .llcrypt unregistered [ 747.639644] Key type ._llcrypt unregistered [ 758.809137] Key type ._llcrypt registered [ 758.812392] Key type .llcrypt registered [ 760.112687] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 11:06:42 (1788188802) [ 761.720375] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg351-server [ 763.301195] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 11:06:45 (1788188805) [ 763.925305] Key type .llcrypt unregistered [ 763.930701] Key type ._llcrypt unregistered [ 769.223203] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 769.476582] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 769.486197] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 769.496189] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 771.996401] Key type ._llcrypt registered [ 771.998656] Key type .llcrypt registered [ 772.197724] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 772.222121] alg: No test for adler32 (adler32-zlib) [ 773.173525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 773.279180] LNet: Added LNI 192.168.203.51@tcp1 [8/256/0/180] [ 774.360981] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 776.780687] LNet: 42867:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 776.787214] LNetError: 42867:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 776.795328] LNet: Removed LNI 192.168.203.51@tcp1 [ 776.807129] LNet: Removed LNI 10.1.2.3@tcp2 [ 777.608806] Key type .llcrypt unregistered [ 777.615092] Key type ._llcrypt unregistered [ 792.934367] Key type ._llcrypt registered [ 792.938935] Key type .llcrypt registered [ 794.620377] Lustre: DEBUG MARKER: == sanity-lnet test 113: Test large skb Netlink dump with many LNet nets (LU-20221) ========================================================== 11:07:16 (1788188836) [ 795.372405] Key type .llcrypt unregistered [ 795.374518] Key type ._llcrypt unregistered [ 801.856746] Key type ._llcrypt registered [ 801.858826] Key type .llcrypt registered [ 802.023363] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 802.042593] alg: No test for adler32 (adler32-zlib) [ 802.976649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 803.019869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 803.037792] LNet: Added LNI 192.168.203.51@tcp1 [8/256/0/180] [ 803.628613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 803.643402] LNet: Added LNI 192.168.203.51@tcp2 [8/256/0/180] [ 804.192869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp3 --if ens2 [ 804.209445] LNet: Added LNI 192.168.203.51@tcp3 [8/256/0/180] [ 805.236853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp4 --if ens2 [ 805.259451] LNet: Added LNI 192.168.203.51@tcp4 [8/256/0/180] [ 806.455727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp5 --if ens2 [ 806.482311] LNet: Added LNI 192.168.203.51@tcp5 [8/256/0/180] [ 807.572372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp6 --if ens2 [ 807.590270] LNet: Added LNI 192.168.203.51@tcp6 [8/256/0/180] [ 809.211532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp7 --if ens2 [ 809.234175] LNet: Added LNI 192.168.203.51@tcp7 [8/256/0/180] [ 811.200342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp8 --if ens2 [ 811.237962] LNet: Added LNI 192.168.203.51@tcp8 [8/256/0/180] [ 813.262313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp9 --if ens2 [ 813.295223] LNet: Added LNI 192.168.203.51@tcp9 [8/256/0/180] [ 815.506635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [ 815.541348] LNet: Added LNI 192.168.203.51@tcp10 [8/256/0/180] [ 817.343146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [ 817.364351] LNet: Added LNI 192.168.203.51@tcp11 [8/256/0/180] [ 819.344829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [ 819.380159] LNet: Added LNI 192.168.203.51@tcp12 [8/256/0/180] [ 821.670444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [ 821.712399] LNet: Added LNI 192.168.203.51@tcp13 [8/256/0/180] [ 824.851658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [ 824.882656] LNet: Added LNI 192.168.203.51@tcp14 [8/256/0/180] [ 827.817421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [ 827.882427] LNet: Added LNI 192.168.203.51@tcp15 [8/256/0/180] [ 830.739328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [ 830.762420] LNet: Added LNI 192.168.203.51@tcp16 [8/256/0/180] [ 833.259148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [ 833.278828] LNet: Added LNI 192.168.203.51@tcp17 [8/256/0/180] [ 835.421405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [ 835.437717] LNet: Added LNI 192.168.203.51@tcp18 [8/256/0/180] [ 837.766795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [ 837.824386] LNet: Added LNI 192.168.203.51@tcp19 [8/256/0/180] [ 840.687807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [ 840.720493] LNet: Added LNI 192.168.203.51@tcp20 [8/256/0/180] [ 843.587282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [ 843.607511] LNet: Added LNI 192.168.203.51@tcp21 [8/256/0/180] [ 846.937683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [ 846.974599] LNet: Added LNI 192.168.203.51@tcp22 [8/256/0/180] [ 849.977618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [ 849.996615] LNet: Added LNI 192.168.203.51@tcp23 [8/256/0/180] [ 853.432395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [ 853.464850] LNet: Added LNI 192.168.203.51@tcp24 [8/256/0/180] [ 856.804506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [ 856.824975] LNet: Added LNI 192.168.203.51@tcp25 [8/256/0/180] [ 859.939572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [ 859.962750] LNet: Added LNI 192.168.203.51@tcp26 [8/256/0/180] [ 863.463353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [ 863.483805] LNet: Added LNI 192.168.203.51@tcp27 [8/256/0/180] [ 867.290806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [ 867.320979] LNet: Added LNI 192.168.203.51@tcp28 [8/256/0/180] [ 870.981359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [ 871.013730] LNet: Added LNI 192.168.203.51@tcp29 [8/256/0/180] [ 874.795935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [ 874.813928] LNet: Added LNI 192.168.203.51@tcp30 [8/256/0/180] [ 878.502580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [ 878.539984] LNet: Added LNI 192.168.203.51@tcp31 [8/256/0/180] [ 882.398916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [ 882.423722] LNet: Added LNI 192.168.203.51@tcp32 [8/256/0/180] [ 886.591659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [ 886.629367] LNet: Added LNI 192.168.203.51@tcp33 [8/256/0/180] [ 891.456233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [ 891.480773] LNet: Added LNI 192.168.203.51@tcp34 [8/256/0/180] [ 895.917531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [ 895.935558] LNet: Added LNI 192.168.203.51@tcp35 [8/256/0/180] [ 900.346387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [ 900.361088] LNet: Added LNI 192.168.203.51@tcp36 [8/256/0/180] [ 905.048208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [ 905.063302] LNet: Added LNI 192.168.203.51@tcp37 [8/256/0/180] [ 909.802868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [ 909.831785] LNet: Added LNI 192.168.203.51@tcp38 [8/256/0/180] [ 914.756249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [ 914.775412] LNet: Added LNI 192.168.203.51@tcp39 [8/256/0/180] [ 919.768249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [ 919.799989] LNet: Added LNI 192.168.203.51@tcp40 [8/256/0/180] [ 925.255437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [ 925.274094] LNet: Added LNI 192.168.203.51@tcp41 [8/256/0/180] [ 930.533487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 930.551398] LNet: Added LNI 192.168.203.51@tcp42 [8/256/0/180] [ 935.432542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [ 935.472929] LNet: Added LNI 192.168.203.51@tcp43 [8/256/0/180] [ 940.228259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [ 940.251883] LNet: Added LNI 192.168.203.51@tcp44 [8/256/0/180] [ 945.596543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [ 945.621542] LNet: Added LNI 192.168.203.51@tcp45 [8/256/0/180] [ 950.968470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [ 950.990553] LNet: Added LNI 192.168.203.51@tcp46 [8/256/0/180] [ 956.517907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [ 956.578212] LNet: Added LNI 192.168.203.51@tcp47 [8/256/0/180] [ 962.003953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [ 962.026971] LNet: Added LNI 192.168.203.51@tcp48 [8/256/0/180] [ 967.813518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [ 967.853980] LNet: Added LNI 192.168.203.51@tcp49 [8/256/0/180] [ 975.091551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [ 975.116322] LNet: Added LNI 192.168.203.51@tcp50 [8/256/0/180] [ 981.155572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp51 --if ens2 [ 981.194588] LNet: Added LNI 192.168.203.51@tcp51 [8/256/0/180] [ 986.997306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp52 --if ens2 [ 987.029564] LNet: Added LNI 192.168.203.51@tcp52 [8/256/0/180] [ 993.578529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp53 --if ens2 [ 993.625313] LNet: Added LNI 192.168.203.51@tcp53 [8/256/0/180] [ 999.898942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp54 --if ens2 [ 999.923450] LNet: Added LNI 192.168.203.51@tcp54 [8/256/0/180] [ 1006.218742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp55 --if ens2 [ 1006.240691] LNet: Added LNI 192.168.203.51@tcp55 [8/256/0/180] [ 1012.973030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp56 --if ens2 [ 1012.989719] LNet: Added LNI 192.168.203.51@tcp56 [8/256/0/180] [ 1021.025820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp57 --if ens2 [ 1021.052339] LNet: Added LNI 192.168.203.51@tcp57 [8/256/0/180] [ 1028.940693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp58 --if ens2 [ 1028.967869] LNet: Added LNI 192.168.203.51@tcp58 [8/256/0/180] [ 1036.095668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp59 --if ens2 [ 1036.121048] LNet: Added LNI 192.168.203.51@tcp59 [8/256/0/180] [ 1044.700139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp60 --if ens2 [ 1044.750313] LNet: Added LNI 192.168.203.51@tcp60 [8/256/0/180] [ 1052.631615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp61 --if ens2 [ 1052.663178] LNet: Added LNI 192.168.203.51@tcp61 [8/256/0/180] [ 1061.407790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp62 --if ens2 [ 1061.421825] LNet: Added LNI 192.168.203.51@tcp62 [8/256/0/180] [ 1068.344668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp63 --if ens2 [ 1068.391841] LNet: Added LNI 192.168.203.51@tcp63 [8/256/0/180] [ 1076.956741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp64 --if ens2 [ 1076.987255] LNet: Added LNI 192.168.203.51@tcp64 [8/256/0/180] [ 1085.579576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp65 --if ens2 [ 1085.593168] LNet: Added LNI 192.168.203.51@tcp65 [8/256/0/180] [ 1094.291651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp66 --if ens2 [ 1094.309756] LNet: Added LNI 192.168.203.51@tcp66 [8/256/0/180] [ 1102.917603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp67 --if ens2 [ 1102.954166] LNet: Added LNI 192.168.203.51@tcp67 [8/256/0/180] [ 1111.772294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp68 --if ens2 [ 1111.810928] LNet: Added LNI 192.168.203.51@tcp68 [8/256/0/180] [ 1121.630231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp69 --if ens2 [ 1121.646796] LNet: Added LNI 192.168.203.51@tcp69 [8/256/0/180] [ 1130.240642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp70 --if ens2 [ 1130.283713] LNet: Added LNI 192.168.203.51@tcp70 [8/256/0/180] [ 1137.794794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp71 --if ens2 [ 1137.819731] LNet: Added LNI 192.168.203.51@tcp71 [8/256/0/180] [ 1146.593822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp72 --if ens2 [ 1146.625174] LNet: Added LNI 192.168.203.51@tcp72 [8/256/0/180] [ 1155.320869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp73 --if ens2 [ 1155.353037] LNet: Added LNI 192.168.203.51@tcp73 [8/256/0/180] [ 1163.450210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp74 --if ens2 [ 1163.468546] LNet: Added LNI 192.168.203.51@tcp74 [8/256/0/180] [ 1173.352445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp75 --if ens2 [ 1173.409478] LNet: Added LNI 192.168.203.51@tcp75 [8/256/0/180] [ 1185.001819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp76 --if ens2 [ 1185.020694] LNet: Added LNI 192.168.203.51@tcp76 [8/256/0/180] [ 1194.943563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp77 --if ens2 [ 1194.970208] LNet: Added LNI 192.168.203.51@tcp77 [8/256/0/180] [ 1206.920864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp78 --if ens2 [ 1206.948447] LNet: Added LNI 192.168.203.51@tcp78 [8/256/0/180] [ 1216.969416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp79 --if ens2 [ 1216.991906] LNet: Added LNI 192.168.203.51@tcp79 [8/256/0/180] [ 1226.876813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp80 --if ens2 [ 1226.902074] LNet: Added LNI 192.168.203.51@tcp80 [8/256/0/180] [ 1235.139600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp81 --if ens2 [ 1235.175965] LNet: Added LNI 192.168.203.51@tcp81 [8/256/0/180] [ 1244.706470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp82 --if ens2 [ 1244.730087] LNet: Added LNI 192.168.203.51@tcp82 [8/256/0/180] [ 1253.907918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp83 --if ens2 [ 1253.927570] LNet: Added LNI 192.168.203.51@tcp83 [8/256/0/180] [ 1264.507489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp84 --if ens2 [ 1264.523962] LNet: Added LNI 192.168.203.51@tcp84 [8/256/0/180] [ 1276.069880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp85 --if ens2 [ 1276.089035] LNet: Added LNI 192.168.203.51@tcp85 [8/256/0/180] [ 1286.938945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp86 --if ens2 [ 1286.982410] LNet: Added LNI 192.168.203.51@tcp86 [8/256/0/180] [ 1296.977459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp87 --if ens2 [ 1296.997265] LNet: Added LNI 192.168.203.51@tcp87 [8/256/0/180] [ 1305.664715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp88 --if ens2 [ 1305.698838] LNet: Added LNI 192.168.203.51@tcp88 [8/256/0/180] [ 1314.629899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp89 --if ens2 [ 1314.648539] LNet: Added LNI 192.168.203.51@tcp89 [8/256/0/180] [ 1325.316268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp90 --if ens2 [ 1325.336033] LNet: Added LNI 192.168.203.51@tcp90 [8/256/0/180] [ 1335.674132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp91 --if ens2 [ 1335.726901] LNet: Added LNI 192.168.203.51@tcp91 [8/256/0/180] [ 1345.598942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp92 --if ens2 [ 1345.620880] LNet: Added LNI 192.168.203.51@tcp92 [8/256/0/180] [ 1355.530703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp93 --if ens2 [ 1355.580593] LNet: Added LNI 192.168.203.51@tcp93 [8/256/0/180] [ 1367.036835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp94 --if ens2 [ 1367.055661] LNet: Added LNI 192.168.203.51@tcp94 [8/256/0/180] [ 1379.345972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp95 --if ens2 [ 1379.366565] LNet: Added LNI 192.168.203.51@tcp95 [8/256/0/180] [ 1391.098907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp96 --if ens2 [ 1391.120130] LNet: Added LNI 192.168.203.51@tcp96 [8/256/0/180] [ 1404.534240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp97 --if ens2 [ 1404.572364] LNet: Added LNI 192.168.203.51@tcp97 [8/256/0/180] [ 1417.343798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp98 --if ens2 [ 1417.362626] LNet: Added LNI 192.168.203.51@tcp98 [8/256/0/180] [ 1429.105414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if ens2 [ 1429.134701] LNet: Added LNI 192.168.203.51@tcp99 [8/256/0/180] [ 1441.396644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp100 --if ens2 [ 1441.435778] LNet: Added LNI 192.168.203.51@tcp100 [8/256/0/180] [ 1453.035941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp101 --if ens2 [ 1453.063938] LNet: Added LNI 192.168.203.51@tcp101 [8/256/0/180] [ 1464.417722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp102 --if ens2 [ 1464.440568] LNet: Added LNI 192.168.203.51@tcp102 [8/256/0/180] [ 1475.338506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp103 --if ens2 [ 1475.376373] LNet: Added LNI 192.168.203.51@tcp103 [8/256/0/180] [ 1486.881674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp104 --if ens2 [ 1486.903959] LNet: Added LNI 192.168.203.51@tcp104 [8/256/0/180] [ 1498.673412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp105 --if ens2 [ 1498.691822] LNet: Added LNI 192.168.203.51@tcp105 [8/256/0/180] [ 1508.631992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp106 --if ens2 [ 1508.661141] LNet: Added LNI 192.168.203.51@tcp106 [8/256/0/180] [ 1520.783334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp107 --if ens2 [ 1520.810568] LNet: Added LNI 192.168.203.51@tcp107 [8/256/0/180] [ 1531.959841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp108 --if ens2 [ 1531.982554] LNet: Added LNI 192.168.203.51@tcp108 [8/256/0/180] [ 1543.795682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp109 --if ens2 [ 1543.829274] LNet: Added LNI 192.168.203.51@tcp109 [8/256/0/180] [ 1560.136221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp110 --if ens2 [ 1560.162616] LNet: Added LNI 192.168.203.51@tcp110 [8/256/0/180] [ 1570.891190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp111 --if ens2 [ 1570.927602] LNet: Added LNI 192.168.203.51@tcp111 [8/256/0/180] [ 1582.026656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp112 --if ens2 [ 1582.045557] LNet: Added LNI 192.168.203.51@tcp112 [8/256/0/180] [ 1595.815461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp113 --if ens2 [ 1595.842416] LNet: Added LNI 192.168.203.51@tcp113 [8/256/0/180] [ 1607.926785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp114 --if ens2 [ 1607.971252] LNet: Added LNI 192.168.203.51@tcp114 [8/256/0/180] [ 1619.251948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp115 --if ens2 [ 1619.279711] LNet: Added LNI 192.168.203.51@tcp115 [8/256/0/180] [ 1635.872606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp116 --if ens2 [ 1635.892953] LNet: Added LNI 192.168.203.51@tcp116 [8/256/0/180] [ 1649.403177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp117 --if ens2 [ 1649.434497] LNet: Added LNI 192.168.203.51@tcp117 [8/256/0/180] [ 1663.450414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp118 --if ens2 [ 1663.481823] LNet: Added LNI 192.168.203.51@tcp118 [8/256/0/180] [ 1676.497249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp119 --if ens2 [ 1676.517805] LNet: Added LNI 192.168.203.51@tcp119 [8/256/0/180] [ 1687.529939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp120 --if ens2 [ 1687.560983] LNet: Added LNI 192.168.203.51@tcp120 [8/256/0/180] [ 1699.518028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp121 --if ens2 [ 1699.542927] LNet: Added LNI 192.168.203.51@tcp121 [8/256/0/180] [ 1716.415964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp122 --if ens2 [ 1716.431925] LNet: Added LNI 192.168.203.51@tcp122 [8/256/0/180] [ 1737.908759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp123 --if ens2 [ 1737.956742] LNet: Added LNI 192.168.203.51@tcp123 [8/256/0/180] [ 1759.468796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp124 --if ens2 [ 1759.490337] LNet: Added LNI 192.168.203.51@tcp124 [8/256/0/180] [ 1775.204587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp125 --if ens2 [ 1775.257933] LNet: Added LNI 192.168.203.51@tcp125 [8/256/0/180] [ 1792.663323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp126 --if ens2 [ 1792.696174] LNet: Added LNI 192.168.203.51@tcp126 [8/256/0/180] [ 1807.273293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp127 --if ens2 [ 1807.307294] LNet: Added LNI 192.168.203.51@tcp127 [8/256/0/180] [ 1821.536580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp128 --if ens2 [ 1821.566737] LNet: Added LNI 192.168.203.51@tcp128 [8/256/0/180] [ 1834.871937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp129 --if ens2 [ 1834.895712] LNet: Added LNI 192.168.203.51@tcp129 [8/256/0/180] [ 1851.252560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp130 --if ens2 [ 1851.273460] LNet: Added LNI 192.168.203.51@tcp130 [8/256/0/180] [ 1865.215682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp131 --if ens2 [ 1865.248240] LNet: Added LNI 192.168.203.51@tcp131 [8/256/0/180] [ 1881.432971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp132 --if ens2 [ 1881.456374] LNet: Added LNI 192.168.203.51@tcp132 [8/256/0/180] [ 1896.711252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp133 --if ens2 [ 1896.738705] LNet: Added LNI 192.168.203.51@tcp133 [8/256/0/180] [ 1911.908744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp134 --if ens2 [ 1911.946544] LNet: Added LNI 192.168.203.51@tcp134 [8/256/0/180] [ 1927.107459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp135 --if ens2 [ 1927.123865] LNet: Added LNI 192.168.203.51@tcp135 [8/256/0/180] [ 1940.163894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp136 --if ens2 [ 1940.180055] LNet: Added LNI 192.168.203.51@tcp136 [8/256/0/180] [ 1957.855637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp137 --if ens2 [ 1957.873770] LNet: Added LNI 192.168.203.51@tcp137 [8/256/0/180] [ 1974.431630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp138 --if ens2 [ 1974.452367] LNet: Added LNI 192.168.203.51@tcp138 [8/256/0/180] [ 1989.133529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp139 --if ens2 [ 1989.168803] LNet: Added LNI 192.168.203.51@tcp139 [8/256/0/180] [ 2002.736733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp140 --if ens2 [ 2002.753819] LNet: Added LNI 192.168.203.51@tcp140 [8/256/0/180] [ 2017.211398] LustreError: 166843:0:(udsp.c:674:lnet_udsp_apply_single_policy()) Bad action type. Expected 2 got 1 [ 2017.707856] LustreError: 166871:0:(udsp.c:674:lnet_udsp_apply_single_policy()) Bad action type. Expected 2 got 1 [ 2017.716286] LustreError: 166871:0:(udsp.c:674:lnet_udsp_apply_single_policy()) Skipped 144 previous similar messages [ 2018.242717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp1 --if ens2 [ 2018.267709] LNet: Removed LNI 192.168.203.51@tcp1 [ 2018.286334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp2 --if ens2 [ 2018.316667] LNet: Removed LNI 192.168.203.51@tcp2 [ 2018.338643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp3 --if ens2 [ 2018.379158] LNet: Removed LNI 192.168.203.51@tcp3 [ 2018.410977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp4 --if ens2 [ 2018.435106] LNet: Removed LNI 192.168.203.51@tcp4 [ 2018.467495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp5 --if ens2 [ 2018.500370] LNet: Removed LNI 192.168.203.51@tcp5 [ 2018.529394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp6 --if ens2 [ 2018.548492] LNet: Removed LNI 192.168.203.51@tcp6 [ 2018.561566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp7 --if ens2 [ 2018.581416] LNet: Removed LNI 192.168.203.51@tcp7 [ 2018.599740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp8 --if ens2 [ 2018.641457] LNet: Removed LNI 192.168.203.51@tcp8 [ 2018.659844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp9 --if ens2 [ 2018.699432] LNet: Removed LNI 192.168.203.51@tcp9 [ 2018.727277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp10 --if ens2 [ 2018.783899] LNet: Removed LNI 192.168.203.51@tcp10 [ 2018.806740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp11 --if ens2 [ 2018.837187] LNet: Removed LNI 192.168.203.51@tcp11 [ 2018.860076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp12 --if ens2 [ 2018.907539] LNet: Removed LNI 192.168.203.51@tcp12 [ 2018.955838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp13 --if ens2 [ 2018.995909] LNet: Removed LNI 192.168.203.51@tcp13 [ 2019.032293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp14 --if ens2 [ 2019.101904] LNet: Removed LNI 192.168.203.51@tcp14 [ 2019.130489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp15 --if ens2 [ 2019.153524] LNet: Removed LNI 192.168.203.51@tcp15 [ 2019.196134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp16 --if ens2 [ 2019.216977] LNet: Removed LNI 192.168.203.51@tcp16 [ 2019.234229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp17 --if ens2 [ 2019.284731] LNet: Removed LNI 192.168.203.51@tcp17 [ 2019.310367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp18 --if ens2 [ 2019.331454] LNet: Removed LNI 192.168.203.51@tcp18 [ 2019.354376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp19 --if ens2 [ 2019.374526] LNet: Removed LNI 192.168.203.51@tcp19 [ 2019.396924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp20 --if ens2 [ 2019.419264] LNet: Removed LNI 192.168.203.51@tcp20 [ 2019.439237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp21 --if ens2 [ 2019.464557] LNet: Removed LNI 192.168.203.51@tcp21 [ 2019.484265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp22 --if ens2 [ 2019.500989] LNet: Removed LNI 192.168.203.51@tcp22 [ 2019.524832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp23 --if ens2 [ 2019.543939] LNet: Removed LNI 192.168.203.51@tcp23 [ 2019.564382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp24 --if ens2 [ 2019.584864] LNet: Removed LNI 192.168.203.51@tcp24 [ 2019.604813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp25 --if ens2 [ 2019.623447] LNet: Removed LNI 192.168.203.51@tcp25 [ 2019.651488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp26 --if ens2 [ 2019.686724] LNet: Removed LNI 192.168.203.51@tcp26 [ 2019.712610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp27 --if ens2 [ 2019.747481] LNet: Removed LNI 192.168.203.51@tcp27 [ 2019.779671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp28 --if ens2 [ 2019.793740] LNet: Removed LNI 192.168.203.51@tcp28 [ 2019.806554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp29 --if ens2 [ 2019.831862] LNet: Removed LNI 192.168.203.51@tcp29 [ 2019.846816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp30 --if ens2 [ 2019.869820] LNet: Removed LNI 192.168.203.51@tcp30 [ 2019.893767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp31 --if ens2 [ 2019.912047] LNet: Removed LNI 192.168.203.51@tcp31 [ 2019.928215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp32 --if ens2 [ 2019.956462] LNet: Removed LNI 192.168.203.51@tcp32 [ 2019.976971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp33 --if ens2 [ 2020.010456] LNet: Removed LNI 192.168.203.51@tcp33 [ 2020.033131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp34 --if ens2 [ 2020.066299] LNet: Removed LNI 192.168.203.51@tcp34 [ 2020.096897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp35 --if ens2 [ 2020.139207] LNet: Removed LNI 192.168.203.51@tcp35 [ 2020.164396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp36 --if ens2 [ 2020.196464] LNet: Removed LNI 192.168.203.51@tcp36 [ 2020.220786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp37 --if ens2 [ 2020.255870] LNet: Removed LNI 192.168.203.51@tcp37 [ 2020.295550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp38 --if ens2 [ 2020.320271] LNet: Removed LNI 192.168.203.51@tcp38 [ 2020.342600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp39 --if ens2 [ 2020.373270] LNet: Removed LNI 192.168.203.51@tcp39 [ 2020.418235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp40 --if ens2 [ 2020.456091] LNet: Removed LNI 192.168.203.51@tcp40 [ 2020.496797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp41 --if ens2 [ 2020.547766] LNet: Removed LNI 192.168.203.51@tcp41 [ 2020.576895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp42 --if ens2 [ 2020.611481] LNet: Removed LNI 192.168.203.51@tcp42 [ 2020.641650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp43 --if ens2 [ 2020.668475] LNet: Removed LNI 192.168.203.51@tcp43 [ 2020.691162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp44 --if ens2 [ 2020.719534] LNet: Removed LNI 192.168.203.51@tcp44 [ 2020.757374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp45 --if ens2 [ 2020.792984] LNet: Removed LNI 192.168.203.51@tcp45 [ 2020.831268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp46 --if ens2 [ 2020.872312] LNet: Removed LNI 192.168.203.51@tcp46 [ 2020.896286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp47 --if ens2 [ 2020.924399] LNet: Removed LNI 192.168.203.51@tcp47 [ 2020.940921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp48 --if ens2 [ 2020.962193] LNet: Removed LNI 192.168.203.51@tcp48 [ 2020.978080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp49 --if ens2 [ 2020.998931] LNet: Removed LNI 192.168.203.51@tcp49 [ 2021.010793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp50 --if ens2 [ 2021.029184] LNet: Removed LNI 192.168.203.51@tcp50 [ 2021.040915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp51 --if ens2 [ 2021.055930] LNet: Removed LNI 192.168.203.51@tcp51 [ 2021.067936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp52 --if ens2 [ 2021.083743] LNet: Removed LNI 192.168.203.51@tcp52 [ 2021.096548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp53 --if ens2 [ 2021.112531] LNet: Removed LNI 192.168.203.51@tcp53 [ 2021.129284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp54 --if ens2 [ 2021.145959] LNet: Removed LNI 192.168.203.51@tcp54 [ 2021.158978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp55 --if ens2 [ 2021.174919] LNet: Removed LNI 192.168.203.51@tcp55 [ 2021.190322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp56 --if ens2 [ 2021.204676] LNet: Removed LNI 192.168.203.51@tcp56 [ 2021.218340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp57 --if ens2 [ 2021.234701] LNet: Removed LNI 192.168.203.51@tcp57 [ 2021.248402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp58 --if ens2 [ 2021.263785] LNet: Removed LNI 192.168.203.51@tcp58 [ 2021.277660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp59 --if ens2 [ 2021.294050] LNet: Removed LNI 192.168.203.51@tcp59 [ 2021.305268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp60 --if ens2 [ 2021.322389] LNet: Removed LNI 192.168.203.51@tcp60 [ 2021.333901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp61 --if ens2 [ 2021.348317] LNet: Removed LNI 192.168.203.51@tcp61 [ 2021.361460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp62 --if ens2 [ 2021.379232] LNet: Removed LNI 192.168.203.51@tcp62 [ 2021.391526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp63 --if ens2 [ 2021.407089] LNet: Removed LNI 192.168.203.51@tcp63 [ 2021.424396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp64 --if ens2 [ 2021.444496] LNet: Removed LNI 192.168.203.51@tcp64 [ 2021.460233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp65 --if ens2 [ 2021.478712] LNet: Removed LNI 192.168.203.51@tcp65 [ 2021.494375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp66 --if ens2 [ 2021.522442] LNet: Removed LNI 192.168.203.51@tcp66 [ 2021.549513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp67 --if ens2 [ 2021.576672] LNet: Removed LNI 192.168.203.51@tcp67 [ 2021.594843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp68 --if ens2 [ 2021.616407] LNet: Removed LNI 192.168.203.51@tcp68 [ 2021.633496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp69 --if ens2 [ 2021.659973] LNet: Removed LNI 192.168.203.51@tcp69 [ 2021.686942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp70 --if ens2 [ 2021.727761] LNet: Removed LNI 192.168.203.51@tcp70 [ 2021.753991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp71 --if ens2 [ 2021.797398] LNet: Removed LNI 192.168.203.51@tcp71 [ 2021.831491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp72 --if ens2 [ 2021.853408] LNet: Removed LNI 192.168.203.51@tcp72 [ 2021.873296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp73 --if ens2 [ 2021.918366] LNet: Removed LNI 192.168.203.51@tcp73 [ 2021.956970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp74 --if ens2 [ 2021.992207] LNet: Removed LNI 192.168.203.51@tcp74 [ 2022.016924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp75 --if ens2 [ 2022.039750] LNet: Removed LNI 192.168.203.51@tcp75 [ 2022.054743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp76 --if ens2 [ 2022.093785] LNet: Removed LNI 192.168.203.51@tcp76 [ 2022.112828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp77 --if ens2 [ 2022.144826] LNet: Removed LNI 192.168.203.51@tcp77 [ 2022.180256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp78 --if ens2 [ 2022.202495] LNet: Removed LNI 192.168.203.51@tcp78 [ 2022.240437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp79 --if ens2 [ 2022.274548] LNet: Removed LNI 192.168.203.51@tcp79 [ 2022.289610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp80 --if ens2 [ 2022.308497] LNet: Removed LNI 192.168.203.51@tcp80 [ 2022.329337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp81 --if ens2 [ 2022.355489] LNet: Removed LNI 192.168.203.51@tcp81 [ 2022.371714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp82 --if ens2 [ 2022.391867] LNet: Removed LNI 192.168.203.51@tcp82 [ 2022.410355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp83 --if ens2 [ 2022.441698] LNet: Removed LNI 192.168.203.51@tcp83 [ 2022.458905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp84 --if ens2 [ 2022.484691] LNet: Removed LNI 192.168.203.51@tcp84 [ 2022.497868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp85 --if ens2 [ 2022.520825] LNet: Removed LNI 192.168.203.51@tcp85 [ 2022.540986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp86 --if ens2 [ 2022.565887] LNet: Removed LNI 192.168.203.51@tcp86 [ 2022.587514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp87 --if ens2 [ 2022.609607] LNet: Removed LNI 192.168.203.51@tcp87 [ 2022.630349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp88 --if ens2 [ 2022.659838] LNet: Removed LNI 192.168.203.51@tcp88 [ 2022.693886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp89 --if ens2 [ 2022.713649] LNet: Removed LNI 192.168.203.51@tcp89 [ 2022.729700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp90 --if ens2 [ 2022.741995] LNet: Removed LNI 192.168.203.51@tcp90 [ 2022.758702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp91 --if ens2 [ 2022.778990] LNet: Removed LNI 192.168.203.51@tcp91 [ 2022.797528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp92 --if ens2 [ 2022.812851] LNet: Removed LNI 192.168.203.51@tcp92 [ 2022.825936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp93 --if ens2 [ 2022.841914] LNet: Removed LNI 192.168.203.51@tcp93 [ 2022.853824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp94 --if ens2 [ 2022.871178] LNet: Removed LNI 192.168.203.51@tcp94 [ 2022.882679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp95 --if ens2 [ 2022.901983] LNet: Removed LNI 192.168.203.51@tcp95 [ 2022.910798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp96 --if ens2 [ 2022.931443] LNet: Removed LNI 192.168.203.51@tcp96 [ 2022.954517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp97 --if ens2 [ 2022.973060] LNet: Removed LNI 192.168.203.51@tcp97 [ 2022.990758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp98 --if ens2 [ 2023.010400] LNet: Removed LNI 192.168.203.51@tcp98 [ 2023.033925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp99 --if ens2 [ 2023.050741] LNet: Removed LNI 192.168.203.51@tcp99 [ 2023.069225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp100 --if ens2 [ 2023.126402] LNet: Removed LNI 192.168.203.51@tcp100 [ 2023.145186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp101 --if ens2 [ 2023.166383] LNet: Removed LNI 192.168.203.51@tcp101 [ 2023.188197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp102 --if ens2 [ 2023.203152] LNet: Removed LNI 192.168.203.51@tcp102 [ 2023.229688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp103 --if ens2 [ 2023.262650] LNet: Removed LNI 192.168.203.51@tcp103 [ 2023.280674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp104 --if ens2 [ 2023.298773] LNet: Removed LNI 192.168.203.51@tcp104 [ 2023.316976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp105 --if ens2 [ 2023.334723] LNet: Removed LNI 192.168.203.51@tcp105 [ 2023.346721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp106 --if ens2 [ 2023.373154] LNet: Removed LNI 192.168.203.51@tcp106 [ 2023.398969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp107 --if ens2 [ 2023.429803] LNet: Removed LNI 192.168.203.51@tcp107 [ 2023.453189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp108 --if ens2 [ 2023.471209] LNet: Removed LNI 192.168.203.51@tcp108 [ 2023.487633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp109 --if ens2 [ 2023.526542] LNet: Removed LNI 192.168.203.51@tcp109 [ 2023.553334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp110 --if ens2 [ 2023.574235] LNet: Removed LNI 192.168.203.51@tcp110 [ 2023.609536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp111 --if ens2 [ 2023.658376] LNet: Removed LNI 192.168.203.51@tcp111 [ 2023.681335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp112 --if ens2 [ 2023.719159] LNet: Removed LNI 192.168.203.51@tcp112 [ 2023.737444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp113 --if ens2 [ 2023.765674] LNet: Removed LNI 192.168.203.51@tcp113 [ 2023.786667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp114 --if ens2 [ 2023.813465] LNet: Removed LNI 192.168.203.51@tcp114 [ 2023.831487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp115 --if ens2 [ 2023.863790] LNet: Removed LNI 192.168.203.51@tcp115 [ 2023.881385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp116 --if ens2 [ 2023.894701] LNet: Removed LNI 192.168.203.51@tcp116 [ 2023.914959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp117 --if ens2 [ 2023.939087] LNet: Removed LNI 192.168.203.51@tcp117 [ 2023.975490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp118 --if ens2 [ 2024.005440] LNet: Removed LNI 192.168.203.51@tcp118 [ 2024.019118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp119 --if ens2 [ 2024.034240] LNet: Removed LNI 192.168.203.51@tcp119 [ 2024.047411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp120 --if ens2 [ 2024.070522] LNet: Removed LNI 192.168.203.51@tcp120 [ 2024.086484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp121 --if ens2 [ 2024.108957] LNet: Removed LNI 192.168.203.51@tcp121 [ 2024.123875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp122 --if ens2 [ 2024.143585] LNet: Removed LNI 192.168.203.51@tcp122 [ 2024.171870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp123 --if ens2 [ 2024.199790] LNet: Removed LNI 192.168.203.51@tcp123 [ 2024.214641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp124 --if ens2 [ 2024.243967] LNet: Removed LNI 192.168.203.51@tcp124 [ 2024.264250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp125 --if ens2 [ 2024.299433] LNet: Removed LNI 192.168.203.51@tcp125 [ 2024.327644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp126 --if ens2 [ 2024.349454] LNet: Removed LNI 192.168.203.51@tcp126 [ 2024.364363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp127 --if ens2 [ 2024.390073] LNet: Removed LNI 192.168.203.51@tcp127 [ 2024.412244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp128 --if ens2 [ 2024.454246] LNet: Removed LNI 192.168.203.51@tcp128 [ 2024.477915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp129 --if ens2 [ 2024.496273] LNet: Removed LNI 192.168.203.51@tcp129 [ 2024.510654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp130 --if ens2 [ 2024.530695] LNet: Removed LNI 192.168.203.51@tcp130 [ 2024.547416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp131 --if ens2 [ 2024.580088] LNet: Removed LNI 192.168.203.51@tcp131 [ 2024.607442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp132 --if ens2 [ 2024.660272] LNet: Removed LNI 192.168.203.51@tcp132 [ 2024.685451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp133 --if ens2 [ 2024.707845] LNet: Removed LNI 192.168.203.51@tcp133 [ 2024.731219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp134 --if ens2 [ 2024.793721] LNet: Removed LNI 192.168.203.51@tcp134 [ 2024.827581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp135 --if ens2 [ 2024.865583] LNet: Removed LNI 192.168.203.51@tcp135 [ 2024.887494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp136 --if ens2 [ 2024.929497] LNet: Removed LNI 192.168.203.51@tcp136 [ 2024.949376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp137 --if ens2 [ 2024.980505] LNet: Removed LNI 192.168.203.51@tcp137 [ 2025.000489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp138 --if ens2 [ 2025.033929] LNet: Removed LNI 192.168.203.51@tcp138 [ 2025.048720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp139 --if ens2 [ 2025.066955] LNet: Removed LNI 192.168.203.51@tcp139 [ 2025.088778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp140 --if ens2 [ 2025.138915] LNet: Removed LNI 192.168.203.51@tcp140 [ 2025.217875] LNet: 167176:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2025.906877] Key type .llcrypt unregistered [ 2025.915810] Key type ._llcrypt unregistered [ 2036.284620] Key type ._llcrypt registered [ 2036.286851] Key type .llcrypt registered [ 2037.583384] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 11:27:59 (1788190079) [ 2038.014216] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2038.026753] alg: No test for adler32 (adler32-zlib) [ 2038.923158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2039.010265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2039.040695] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 2040.043938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2040.066748] LNet: 168132:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2040.074281] LNetError: 168132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2040.090549] LNet: Removed LNI 192.168.203.51@tcp [ 2040.108057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2040.189870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 2040.249867] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 2045.989731] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 11:28:08 (1788190088) [ 2046.101853] LNet: 168716:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2046.109885] LNetError: 168716:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2046.129368] LNet: Removed LNI 192.168.203.51@tcp [ 2046.754400] Key type .llcrypt unregistered [ 2046.757197] Key type ._llcrypt unregistered [ 2052.224126] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2052.370498] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2052.386256] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2052.393674] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2054.851995] Key type ._llcrypt registered [ 2054.854253] Key type .llcrypt registered [ 2055.051209] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2055.066506] alg: No test for adler32 (adler32-zlib) [ 2056.006512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2056.075030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2056.100253] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 2056.951801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2056.969949] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2058.871901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2058.892588] LNet: 169462:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2058.896617] LNetError: 169462:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2058.900750] LNet: Removed LNI 10.1.2.3@tcp [ 2058.910074] LNet: Removed LNI 192.168.203.51@tcp [ 2058.925483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2058.961984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 2058.980283] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 2058.983873] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2059.348152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2059.363992] LNet: Removed LNI 10.1.2.3@tcp [ 2059.375919] LNet: Removed LNI 192.168.203.51@tcp [ 2059.403733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2059.459355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 2059.483140] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 2059.489542] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2059.937830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2059.959640] LNet: Added LNI 192.168.203.51@tcp2 [8/256/0/180] [ 2060.835279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2060.861748] LNet: 169651:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2060.869583] LNet: 169651:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2060.876125] LNetError: 169651:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2060.887108] LNetError: 169651:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2060.902925] LNet: Removed LNI 10.1.2.3@tcp [ 2060.905525] LNet: Removed LNI 192.168.203.51@tcp [ 2060.926454] LNet: Removed LNI 192.168.203.51@tcp2 [ 2060.970582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2061.044963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 2061.075116] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 2061.082223] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2061.091986] LNet: Added LNI 192.168.203.51@tcp2 [8/256/0/180] [ 2067.201426] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 11:28:29 (1788190109) [ 2067.429985] LNet: 170250:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2067.441533] LNetError: 170250:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2067.447479] LNetError: 170250:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2067.454091] LNet: Removed LNI 10.1.2.3@tcp [ 2067.456545] LNet: Removed LNI 192.168.203.51@tcp [ 2067.478933] LNet: Removed LNI 192.168.203.51@tcp2 [ 2068.079764] Key type .llcrypt unregistered [ 2068.082053] Key type ._llcrypt unregistered [ 2074.404376] Key type ._llcrypt registered [ 2074.408113] Key type .llcrypt registered [ 2074.652162] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2074.684689] alg: No test for adler32 (adler32-zlib) [ 2075.615922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2075.693800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2075.744146] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 2076.698398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2076.718710] LNet: 170642:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2076.727039] LNetError: 170642:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2076.748611] LNet: Removed LNI 192.168.203.51@tcp [ 2076.780519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2076.851315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 2076.876609] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 2077.498408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2077.515410] LNet: 170672:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2077.527334] LNetError: 170672:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2077.550937] LNet: Removed LNI 192.168.203.51@tcp [ 2077.598967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2077.681321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 2077.717597] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 2078.307079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2078.341278] LNet: Removed LNI 192.168.203.51@tcp [ 2078.366149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2078.413071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 2078.433984] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 2078.893854] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2079.115976] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2079.129297] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2079.137473] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2081.488528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2081.514599] LNet: 170774:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2081.518586] LNet: 170774:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2081.534765] LNetError: 170774:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2081.542420] LNetError: 170774:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2081.556561] LNet: Removed LNI 192.168.203.51@tcp [ 2081.578868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2081.635581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 2081.656888] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 2088.259388] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 11:28:50 (1788190130) [ 2088.371808] LNet: 171364:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2088.397216] LNetError: 171364:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2088.421536] LNet: Removed LNI 192.168.203.51@tcp [ 2089.233746] Key type .llcrypt unregistered [ 2089.236962] Key type ._llcrypt unregistered [ 2096.422459] Key type ._llcrypt registered [ 2096.427470] Key type .llcrypt registered [ 2096.661724] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2096.675981] alg: No test for adler32 (adler32-zlib) [ 2097.681476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2097.801749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2097.817777] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 2099.206821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2099.258456] LNet: 171768:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2099.273563] LNetError: 171768:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2099.324251] LNet: Removed LNI 192.168.203.51@tcp [ 2099.355770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2099.519628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 2099.564670] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 2109.284087] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 11:29:10 (1788190150) [ 2109.551069] LNet: 172356:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2109.572604] LNetError: 172356:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2109.606819] LNet: Removed LNI 192.168.203.51@tcp [ 2110.693280] Key type .llcrypt unregistered [ 2110.695897] Key type ._llcrypt unregistered [ 2119.348549] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2119.515256] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2119.527417] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2119.534203] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2122.149099] Key type ._llcrypt registered [ 2122.150637] Key type .llcrypt registered [ 2122.398081] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2122.414405] alg: No test for adler32 (adler32-zlib) [ 2123.453642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2123.515368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2123.540177] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 2124.355507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2124.376132] LNet: 172808:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2124.380945] LNetError: 172808:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2124.405609] LNet: Removed LNI 192.168.203.51@tcp [ 2124.425528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2124.487245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 2124.524151] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 2131.631118] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 11:29:34 (1788190174) [ 2131.886053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2131.921865] LNet: 173406:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2131.940770] LNetError: 173406:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2131.964261] LNet: Removed LNI 192.168.203.51@tcp [ 2131.975703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2132.069143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 2138.674750] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 11:29:40 (1788190180) [ 2138.755991] LNet: 173979:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2139.502285] Key type .llcrypt unregistered [ 2139.506323] Key type ._llcrypt unregistered [ 2145.685874] Key type ._llcrypt registered [ 2145.690844] Key type .llcrypt registered [ 2145.958507] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2146.004418] alg: No test for adler32 (adler32-zlib) [ 2147.009633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2147.109426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2147.151315] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 2148.323949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2148.349928] LNet: 174383:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2148.358643] LNetError: 174383:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2148.394642] LNet: Removed LNI 192.168.203.51@tcp [ 2148.431793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2148.540623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 2148.593975] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 2155.977891] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 11:29:58 (1788190198) [ 2156.162980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2156.238242] LNet: 174969:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2156.264871] LNetError: 174969:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2156.282192] LNet: Removed LNI 192.168.203.51@tcp [ 2156.306702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2156.361753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 2162.494570] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 11:30:04 (1788190204) [ 2163.896682] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 2165.665809] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 11:30:08 (1788190208) [ 2167.003950] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 2168.521197] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 11:30:10 (1788190210) [ 2170.442272] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 2171.747587] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 11:30:14 (1788190214) [ 2173.173926] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 2174.541242] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 11:30:17 (1788190217) [ 2175.925643] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 2177.943110] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 11:30:20 (1788190220) [ 2178.109331] LNet: 177022:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2178.707814] Key type .llcrypt unregistered [ 2178.711074] Key type ._llcrypt unregistered [ 2184.309622] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2184.468427] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2184.481963] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2184.493765] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2186.936379] Key type ._llcrypt registered [ 2186.940238] Key type .llcrypt registered [ 2187.105996] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2187.129216] alg: No test for adler32 (adler32-zlib) [ 2188.033913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2188.101991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 2188.122262] LNet: Added LNI 192.168.203.51@tcp [8/512/0/180] [ 2188.131309] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 2188.133216] LNet: 177345:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 2188.222752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2188.245297] LNet: 177364:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2188.252740] LNetError: 177364:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2188.261860] LNet: Removed LNI 10.1.2.3@tcp [ 2188.269700] LNetError: 177364:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2188.285237] LNet: Removed LNI 192.168.203.51@tcp [ 2188.319888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2188.375559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 2188.399508] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 2188.469545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2188.487903] LNet: 177388:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2188.493427] LNetError: 177388:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2188.510946] LNet: Removed LNI 192.168.203.51@tcp [ 2188.527244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2188.579570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 2188.607076] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 2188.611085] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2188.613242] LNet: 177396:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 2188.697273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2188.712315] LNet: 177414:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2188.722613] LNetError: 177414:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2188.731071] LNet: Removed LNI 10.1.2.3@tcp [ 2188.734957] LNetError: 177414:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2188.755852] LNet: Removed LNI 192.168.203.51@tcp [ 2188.774406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2188.830887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 2188.847726] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 2188.850554] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2188.852026] LNet: 177422:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 2189.023324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2189.043189] LNet: 177448:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2189.050791] LNetError: 177448:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2189.057757] LNet: Removed LNI 10.1.2.3@tcp [ 2189.060679] LNetError: 177448:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2189.071257] LNet: Removed LNI 192.168.203.51@tcp [ 2189.105503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2189.219148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 2189.261452] LNet: Added LNI 192.168.203.51@tcp [8/256/0/182] [ 2189.265274] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 2189.365589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2189.381426] LNet: 177476:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2189.394455] LNetError: 177476:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2189.397581] LNet: Removed LNI 10.1.2.3@tcp [ 2189.400810] LNetError: 177476:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2189.425582] LNet: Removed LNI 192.168.203.51@tcp [ 2189.450695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2189.494785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 2189.511237] LNet: Added LNI 192.168.203.51@tcp [8/256/0/182] [ 2189.514818] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 2189.517918] LNet: 177484:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 2189.599958] LNet: Removed LNI 10.1.2.3@tcp [ 2189.617070] LNet: Removed LNI 192.168.203.51@tcp [ 2190.396503] Key type .llcrypt unregistered [ 2190.403767] Key type ._llcrypt unregistered [ 2200.698307] Key type ._llcrypt registered [ 2200.700415] Key type .llcrypt registered [ 2201.937203] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 11:30:44 (1788190244) [ 2203.399169] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 2205.035312] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 11:30:47 (1788190247) [ 2205.431533] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2205.449053] alg: No test for adler32 (adler32-zlib) [ 2206.433757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2206.502110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2206.536568] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 2207.218170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.203.51 [ 2207.239758] LNet: 178747:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 2207.250308] LNet: Removed LNI 192.168.203.51@tcp [ 2212.950111] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 11:30:55 (1788190255) [ 2213.094970] LNet: 179310:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2213.108790] LNetError: 179310:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2213.135847] LNet: Removed LNI 192.168.203.51@tcp [ 2213.692362] Key type .llcrypt unregistered [ 2213.698480] Key type ._llcrypt unregistered [ 2219.665526] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2219.900627] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2222.565166] Key type ._llcrypt registered [ 2222.567522] Key type .llcrypt registered [ 2222.729642] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2222.740355] alg: No test for adler32 (adler32-zlib) [ 2224.025260] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2229.900533] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 11:31:12 (1788190272) [ 2230.081362] LNet: 180225:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2230.098325] LNetError: 180225:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2230.116597] LNet: Removed LNI 10.1.2.3@tcp [ 2230.702530] Key type .llcrypt unregistered [ 2230.709591] Key type ._llcrypt unregistered [ 2235.778165] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2235.927638] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2235.932836] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2235.941785] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2238.598853] Key type ._llcrypt registered [ 2238.601735] Key type .llcrypt registered [ 2238.729549] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2238.739960] alg: No test for adler32 (adler32-zlib) [ 2239.630955] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2245.586713] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 11:31:27 (1788190287) [ 2247.694501] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 2249.346398] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 11:31:31 (1788190291) [ 2249.594619] LNet: 181407:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2249.603813] LNetError: 181407:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2249.624812] LNet: Removed LNI 10.1.2.3@tcp [ 2250.303364] Key type .llcrypt unregistered [ 2250.305765] Key type ._llcrypt unregistered [ 2256.763874] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2256.998774] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2257.010418] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2257.021668] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2259.893207] Key type ._llcrypt registered [ 2259.897367] Key type .llcrypt registered [ 2260.198925] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2260.226202] alg: No test for adler32 (adler32-zlib) [ 2261.220983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2261.326302] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2270.434709] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 11:31:52 (1788190312) [ 2271.185798] LNet: 182632:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2271.201411] LNetError: 182632:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2271.225804] LNet: Removed LNI 10.1.2.3@tcp [ 2271.809542] Key type .llcrypt unregistered [ 2271.812475] Key type ._llcrypt unregistered [ 2278.677332] Key type ._llcrypt registered [ 2278.680112] Key type .llcrypt registered [ 2278.969320] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2278.990262] alg: No test for adler32 (adler32-zlib) [ 2279.914930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2279.928974] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 2287.018034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.151@tcp [ 2287.901301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp -m GET -e local_interrupt -r 1 [ 2287.967508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp -m GET -e local_interrupt -r 1 [ 2288.010330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.151@tcp [ 2288.031905] LNet: There was an unexpected network error while writing to 192.168.203.151: rc = -22 [ 2288.167230] LNet: 1 local NIs in recovery (showing 1): 192.168.203.51@tcp [ 2288.264015] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2288.317760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2288.373386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2288.405256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2288.461199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2288.503760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2288.557190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2288.645439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2288.682230] LNet: 183067:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2288.726886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2288.780348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2288.823762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2288.877153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2288.920551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2288.952459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2288.982706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 2289.429901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp -m GET -e local_dropped -r 1 [ 2289.486194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp -m GET -e local_dropped -r 1 [ 2289.512222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.151@tcp [ 2289.527170] LNet: There was an unexpected network error while writing to 192.168.203.151: rc = -22 [ 2289.663873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2289.693617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2289.724970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2289.747064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2289.777101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2289.805144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2289.826623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2289.848428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2289.871133] LNet: 183176:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2289.886922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2289.923342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2289.958866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2289.998837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2290.031626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2290.055491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2290.086420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 2290.526823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp -m GET -e local_aborted -r 1 [ 2290.603223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp -m GET -e local_aborted -r 1 [ 2290.651483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.151@tcp [ 2290.674271] LNet: There was an unexpected network error while writing to 192.168.203.151: rc = -22 [ 2290.988710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2291.037697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2291.093539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2291.136623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2291.161530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2291.188617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2291.216261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2291.242910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2291.261473] LNet: 183286:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2291.280433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2291.314821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2291.347627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2291.376941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2291.455217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2291.500995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2291.528460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 2291.937722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp -m GET -e local_no_route -r 1 [ 2291.996417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp -m GET -e local_no_route -r 1 [ 2292.064434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.151@tcp [ 2292.521692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2292.546894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2292.580367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2292.619863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2292.675315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2292.694770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2292.724599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2292.754897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2292.789638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2292.846643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2292.894556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2292.923841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2292.969715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2293.003694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2293.032581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 2293.447211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp -m GET -e local_timeout -r 1 [ 2293.497208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp -m GET -e local_timeout -r 1 [ 2293.539562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.151@tcp [ 2293.560746] LNet: There was an unexpected network error while writing to 192.168.203.151: rc = -22 [ 2293.564210] LNet: Skipped 2 previous similar messages [ 2293.722962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2293.771799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2293.817917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2293.854723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2293.916237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2293.951294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2293.980314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2294.016887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2294.044577] LNet: 183504:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2294.049447] LNet: 183504:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 2294.096978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2294.169325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2294.206435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2294.240984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2294.276942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2294.307497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2294.332950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 2294.865179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp -m GET -e local_error -r 1 [ 2294.896915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp -m GET -e local_error -r 1 [ 2294.957549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.151@tcp [ 2295.224235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2295.276994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2295.328792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2295.353932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2295.385243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2295.431492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2295.466235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2295.511324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2295.559828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2295.623880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2295.660567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2295.720435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2295.756654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2295.778586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2295.802734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 2296.026484] LNet: 183654:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2296.033526] LNetError: 183654:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2297.068535] LNet: Removed LNI 192.168.203.51@tcp [ 2297.587480] Key type .llcrypt unregistered [ 2297.591743] Key type ._llcrypt unregistered [ 2312.419063] Key type ._llcrypt registered [ 2312.422308] Key type .llcrypt registered [ 2313.802318] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 11:32:35 (1788190355) [ 2315.164226] Key type .llcrypt unregistered [ 2315.169218] Key type ._llcrypt unregistered [ 2323.737416] Key type ._llcrypt registered [ 2323.744684] Key type .llcrypt registered [ 2324.109374] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2324.152937] alg: No test for adler32 (adler32-zlib) [ 2325.168439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2325.220654] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 2334.568219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.151@tcp [ 2345.444462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2345.512785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2345.547596] LNet: Added LNI 192.168.203.51@tcp1 [8/256/0/180] [ 2346.836373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp -m GET -e local_interrupt -r 1 [ 2346.869406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp1 -m GET -e local_interrupt -r 1 [ 2346.896912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp -m GET -e local_interrupt -r 1 [ 2346.924150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp1 -m GET -e local_interrupt -r 1 [ 2346.955690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp -m GET -e local_interrupt -r 1 [ 2346.995972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp1 -m GET -e local_interrupt -r 1 [ 2347.026284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp -m GET -e local_interrupt -r 1 [ 2347.056763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp1 -m GET -e local_interrupt -r 1 [ 2347.107722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.151@tcp [ 2347.146308] LNet: There was an unexpected network error while writing to 192.168.203.151: rc = -22 [ 2347.159365] LNet: 1 local NIs in recovery (showing 1): 192.168.203.51@tcp [ 2347.477163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2347.523177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2347.562514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2347.609517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2347.630636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2347.651651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2347.671466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2347.701481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2347.724302] LNet: 185116:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2347.759247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2347.796624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2347.827117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2347.865826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2347.912098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2347.941064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2347.971543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 2348.404879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp -m GET -e local_dropped -r 1 [ 2348.433963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp1 -m GET -e local_dropped -r 1 [ 2348.472311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp -m GET -e local_dropped -r 1 [ 2348.511737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp1 -m GET -e local_dropped -r 1 [ 2348.553825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp -m GET -e local_dropped -r 1 [ 2348.592952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp1 -m GET -e local_dropped -r 1 [ 2348.653730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp -m GET -e local_dropped -r 1 [ 2348.707100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp1 -m GET -e local_dropped -r 1 [ 2348.755619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.151@tcp [ 2348.774903] LNet: There was an unexpected network error while writing to 192.168.203.151: rc = -22 [ 2348.779671] LNet: Skipped 2 previous similar messages [ 2348.980439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2349.004089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2349.058391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2349.097873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2349.118349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2349.148893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2349.190667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2349.224317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2349.256801] LNet: 185237:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2349.301060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2349.348707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2349.406032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2349.456203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2349.520581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2349.576762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2349.629655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 2350.253622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp -m GET -e local_aborted -r 1 [ 2350.288805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp1 -m GET -e local_aborted -r 1 [ 2350.339202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp -m GET -e local_aborted -r 1 [ 2350.379358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp1 -m GET -e local_aborted -r 1 [ 2350.410559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp -m GET -e local_aborted -r 1 [ 2350.440807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp1 -m GET -e local_aborted -r 1 [ 2350.475416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp -m GET -e local_aborted -r 1 [ 2350.511987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp1 -m GET -e local_aborted -r 1 [ 2350.533153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.151@tcp [ 2350.544773] LNet: There was an unexpected network error while writing to 192.168.203.151: rc = -22 [ 2350.550155] LNet: Skipped 2 previous similar messages [ 2350.783564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2350.817548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2350.849187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2350.893614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2350.927300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2350.958860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2350.988207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2351.010095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2351.022435] LNet: 185358:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2351.034909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2351.074411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2351.136302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2351.211406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2351.287147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2351.357814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2351.458297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 2352.265092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp -m GET -e local_no_route -r 1 [ 2352.330556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp1 -m GET -e local_no_route -r 1 [ 2352.363001] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp -m GET -e local_no_route -r 1 [ 2352.406734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp1 -m GET -e local_no_route -r 1 [ 2352.487393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp -m GET -e local_no_route -r 1 [ 2352.548276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp1 -m GET -e local_no_route -r 1 [ 2352.608910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp -m GET -e local_no_route -r 1 [ 2352.645339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp1 -m GET -e local_no_route -r 1 [ 2352.700815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.151@tcp [ 2352.715166] LNet: There was an unexpected network error while writing to 192.168.203.151: rc = -22 [ 2352.719247] LNet: Skipped 2 previous similar messages [ 2352.976820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2353.016830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2353.058303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2353.089873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2353.126533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2353.161043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2353.216271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2353.260759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2353.295590] LNet: 185480:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2353.326384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2353.358460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2353.392202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2353.448920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2353.487835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2353.528991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2353.574971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 2354.209676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp -m GET -e local_timeout -r 1 [ 2354.261821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp1 -m GET -e local_timeout -r 1 [ 2354.321867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp -m GET -e local_timeout -r 1 [ 2354.365328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp1 -m GET -e local_timeout -r 1 [ 2354.450336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp -m GET -e local_timeout -r 1 [ 2354.507320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp1 -m GET -e local_timeout -r 1 [ 2354.556240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp -m GET -e local_timeout -r 1 [ 2354.619716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp1 -m GET -e local_timeout -r 1 [ 2354.695316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.151@tcp [ 2354.941277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2354.973367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2355.004500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2355.051162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2355.110783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2355.183845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2355.247622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2355.309868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2355.370992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2355.434809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2355.469273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2355.526597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2355.571954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2355.639048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2355.692926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 2356.405874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp -m GET -e local_error -r 1 [ 2356.448722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp1 -m GET -e local_error -r 1 [ 2356.480580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp -m GET -e local_error -r 1 [ 2356.509443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp1 -m GET -e local_error -r 1 [ 2356.535484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp -m GET -e local_error -r 1 [ 2356.567287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp1 -m GET -e local_error -r 1 [ 2356.599320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp -m GET -e local_error -r 1 [ 2356.636538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp1 -m GET -e local_error -r 1 [ 2356.667901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.151@tcp [ 2356.862315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2356.893536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2356.930195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2356.956147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2356.998161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2357.027291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2357.068483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2357.104520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2357.132209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2357.178957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2357.207842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2357.232628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2357.277730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2357.317419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2357.391474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 2362.653659] LNet: 185798:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2362.667365] LNetError: 185798:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2362.684497] LNet: Removed LNI 192.168.203.51@tcp [ 2362.706284] LNet: Removed LNI 192.168.203.51@tcp1 [ 2363.059330] Key type .llcrypt unregistered [ 2363.061927] Key type ._llcrypt unregistered [ 2377.210415] Key type ._llcrypt registered [ 2377.212973] Key type .llcrypt registered [ 2378.501373] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 11:33:40 (1788190420) [ 2379.365453] Key type .llcrypt unregistered [ 2379.370838] Key type ._llcrypt unregistered [ 2387.302337] Key type ._llcrypt registered [ 2387.308529] Key type .llcrypt registered [ 2387.536560] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2387.545525] alg: No test for adler32 (adler32-zlib) [ 2388.592348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2388.637104] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 2398.361746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.151@tcp [ 2399.348946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp -m GET -e remote_dropped -r 1 [ 2399.425592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp -m GET -e remote_dropped -r 1 [ 2399.461050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.151@tcp [ 2399.478139] LNet: There was an unexpected network error while writing to 192.168.203.151: rc = -22 [ 2399.689397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2399.715518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2399.744353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2399.764051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2399.789106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2399.828198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2399.858461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2399.893574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2399.922123] LNet: 187043:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2399.942951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2399.973551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2400.020281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2400.058788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2400.084500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2400.118843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2400.138348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 2400.582893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp -m GET -e remote_error -r 1 [ 2400.637476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp -m GET -e remote_error -r 1 [ 2400.682473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.151@tcp [ 2400.702128] LNet: There was an unexpected network error while writing to 192.168.203.151: rc = -22 [ 2400.940685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2400.986755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2401.026024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2401.061286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2401.113283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2401.158414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2401.209568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2401.244408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2401.273378] LNet: 187152:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2401.295154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2401.340643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2401.390752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2401.410101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2401.453202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2401.501715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2401.597435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 2402.262255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp -m GET -e remote_timeout -r 1 [ 2402.352248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp -m GET -e remote_timeout -r 1 [ 2402.400857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.151@tcp [ 2402.425333] LNet: There was an unexpected network error while writing to 192.168.203.151: rc = -22 [ 2402.622155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2402.648726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2402.688904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2402.731073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2402.789788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2402.832130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2402.877586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2403.003297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2403.037995] LNet: 187261:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2403.051205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2403.070849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2403.087448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2403.119553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2403.177324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2403.251158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2403.335812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 2403.634168] LNet: 187302:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2403.653909] LNetError: 187302:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2403.683469] LNet: Removed LNI 192.168.203.51@tcp [ 2404.273618] Key type .llcrypt unregistered [ 2404.277522] Key type ._llcrypt unregistered [ 2422.914141] Key type ._llcrypt registered [ 2422.916429] Key type .llcrypt registered [ 2424.436632] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 11:34:26 (1788190466) [ 2425.309676] Key type .llcrypt unregistered [ 2425.317035] Key type ._llcrypt unregistered [ 2432.615287] Key type ._llcrypt registered [ 2432.618770] Key type .llcrypt registered [ 2432.847634] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2432.873514] alg: No test for adler32 (adler32-zlib) [ 2433.770309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2433.798710] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 2442.271895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.151@tcp [ 2451.139950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2451.179503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2451.209423] LNet: Added LNI 192.168.203.51@tcp1 [8/256/0/180] [ 2452.462911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp -m GET -e remote_dropped -r 1 [ 2452.491701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp1 -m GET -e remote_dropped -r 1 [ 2452.534737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp -m GET -e remote_dropped -r 1 [ 2452.577111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp1 -m GET -e remote_dropped -r 1 [ 2452.648142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp -m GET -e remote_dropped -r 1 [ 2452.695903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp1 -m GET -e remote_dropped -r 1 [ 2452.727791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp -m GET -e remote_dropped -r 1 [ 2452.753700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp1 -m GET -e remote_dropped -r 1 [ 2452.790314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.151@tcp [ 2452.821359] LNet: There was an unexpected network error while writing to 192.168.203.151: rc = -22 [ 2452.833065] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.151@tcp [ 2453.097668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2453.137589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2453.178851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2453.208698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2453.238607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2453.273680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2453.305232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2453.348283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2453.374810] LNet: 188765:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2453.393377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2453.425460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2453.465784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2453.512123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2453.547487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2453.579237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2453.630299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 2453.707266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2454.281293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp -m GET -e remote_error -r 1 [ 2454.310393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp1 -m GET -e remote_error -r 1 [ 2454.339866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp -m GET -e remote_error -r 1 [ 2454.383272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp1 -m GET -e remote_error -r 1 [ 2454.422925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp -m GET -e remote_error -r 1 [ 2454.452607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp1 -m GET -e remote_error -r 1 [ 2454.486244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp -m GET -e remote_error -r 1 [ 2454.533976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp1 -m GET -e remote_error -r 1 [ 2454.583535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.151@tcp [ 2454.598954] LNet: There was an unexpected network error while writing to 192.168.203.151: rc = -22 [ 2454.607214] LNet: Skipped 2 previous similar messages [ 2454.861372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2454.888826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2454.928446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2454.957371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2454.991385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2455.019994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2455.044795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2455.074728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2455.097843] LNet: 188888:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2455.125742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2455.166490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2455.217995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2455.246641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2455.310712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2455.361563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2455.392956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 2455.453699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2455.998676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp -m GET -e remote_timeout -r 1 [ 2456.026643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp1 -m GET -e remote_timeout -r 1 [ 2456.082433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp -m GET -e remote_timeout -r 1 [ 2456.120274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp1 -m GET -e remote_timeout -r 1 [ 2456.143694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp -m GET -e remote_timeout -r 1 [ 2456.186473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp1 -m GET -e remote_timeout -r 1 [ 2456.246867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp -m GET -e remote_timeout -r 1 [ 2456.305088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp1 -m GET -e remote_timeout -r 1 [ 2456.332778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.151@tcp [ 2456.350655] LNet: There was an unexpected network error while writing to 192.168.203.151: rc = -22 [ 2456.605661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2456.650622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2456.683173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2456.708733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2456.751135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2456.787411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2456.829648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2456.871156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2456.896268] LNet: 189011:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2456.916053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2456.967854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2457.013400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2457.073531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2457.117954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2457.162640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2457.195303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 2457.248531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2462.211511] LNet: 189089:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2462.226717] LNetError: 189089:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2462.239318] LNet: Removed LNI 192.168.203.51@tcp [ 2462.258596] LNet: Removed LNI 192.168.203.51@tcp1 [ 2462.662648] Key type .llcrypt unregistered [ 2462.664192] Key type ._llcrypt unregistered [ 2476.335504] Key type ._llcrypt registered [ 2476.337442] Key type .llcrypt registered [ 2477.797224] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 11:35:19 (1788190519) [ 2479.522843] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 2481.274826] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 11:35:23 (1788190523) [ 2482.239797] Key type .llcrypt unregistered [ 2482.248296] Key type ._llcrypt unregistered [ 2488.331338] Key type ._llcrypt registered [ 2488.333947] Key type .llcrypt registered [ 2488.562639] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2488.578763] alg: No test for adler32 (adler32-zlib) [ 2489.483984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2489.520264] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 2497.122104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.151@tcp [ 2497.715305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp -m GET -e network_timeout -r 1 [ 2497.752298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp -m GET -e network_timeout -r 1 [ 2497.793122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.151@tcp [ 2497.810590] LNet: There was an unexpected network error while writing to 192.168.203.151: rc = -22 [ 2498.048940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2498.081201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2498.110638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2498.147425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2498.175897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2498.211040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2498.256978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2498.298098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2498.320570] LNet: 190628:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2498.336261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2498.360404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2498.405897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2498.457504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2498.491939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2498.533583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2498.791320] LNet: 190667:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2498.803403] LNetError: 190667:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2498.830114] LNet: Removed LNI 192.168.203.51@tcp [ 2499.270652] Key type .llcrypt unregistered [ 2499.279104] Key type ._llcrypt unregistered [ 2515.936163] Key type ._llcrypt registered [ 2515.938516] Key type .llcrypt registered [ 2516.146426] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2516.181325] alg: No test for adler32 (adler32-zlib) [ 2517.168038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2517.216156] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 2524.741972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.151@tcp [ 2532.521199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2532.547982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2532.578216] LNet: Added LNI 192.168.203.51@tcp1 [8/256/0/180] [ 2533.977871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp -m GET -e network_timeout -r 1 [ 2534.023278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp1 -m GET -e network_timeout -r 1 [ 2534.096447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp -m GET -e network_timeout -r 1 [ 2534.151223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp1 -m GET -e network_timeout -r 1 [ 2534.185897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp -m GET -e network_timeout -r 1 [ 2534.234731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp1 -m GET -e network_timeout -r 1 [ 2534.285336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp -m GET -e network_timeout -r 1 [ 2534.332687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp1 -m GET -e network_timeout -r 1 [ 2534.376475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.151@tcp [ 2534.427225] LNet: There was an unexpected network error while writing to 192.168.203.151: rc = -22 [ 2534.626452] LNet: 1 local NIs in recovery (showing 1): 192.168.203.51@tcp [ 2534.639645] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.151@tcp [ 2534.749570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2534.803700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2534.854637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2534.896454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2534.947282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2534.995120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2535.044630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2535.076491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2535.122690] LNet: 191540:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2535.136623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2535.155821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2535.173210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2535.190994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2535.212481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2535.248503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2539.190675] LNet: There was an unexpected network error while writing to 192.168.203.151: rc = -22 [ 2539.475821] LNet: 191614:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2539.481937] LNetError: 191614:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2540.511696] LNet: Removed LNI 192.168.203.51@tcp [ 2540.526496] LNetError: 191614:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2540.545363] LNet: Removed LNI 192.168.203.51@tcp1 [ 2541.011865] Key type .llcrypt unregistered [ 2541.014543] Key type ._llcrypt unregistered [ 2554.182711] Key type ._llcrypt registered [ 2554.185160] Key type .llcrypt registered [ 2555.288398] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 11:36:37 (1788190597) [ 2555.730466] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2555.746312] alg: No test for adler32 (adler32-zlib) [ 2556.692550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2556.801044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2556.829691] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 2557.676331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2557.697455] LNet: Added LNI 192.168.203.51@tcp1 [8/256/0/180] [ 2558.490806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.51@tcp [ 2558.568610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 2558.637908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 2558.675303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 2558.720357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2558.765522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 2558.828311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.51@tcp [ 2558.865563] LNetError: 192712:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.51@tcp to recovery [ 2559.839889] LNet: 1 local NIs in recovery (showing 1): 192.168.203.51@tcp [ 2559.840074] LNet: There was an unexpected network error while writing to 192.168.203.51: rc = -22 [ 2561.887350] LNet: There was an unexpected network error while writing to 192.168.203.51: rc = -22 [ 2562.629729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 2565.983778] LNet: There was an unexpected network error while writing to 192.168.203.51: rc = -22 [ 2566.280933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 2566.329816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 2566.388430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2566.442234] LNet: 192819:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2566.451914] LNetError: 192819:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2566.458349] LNet: Removed LNI 192.168.203.51@tcp [ 2566.466461] LNet: Removed LNI 192.168.203.51@tcp1 [ 2566.484923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2566.535317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2566.569776] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 2567.196583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2567.218739] LNet: Added LNI 192.168.203.51@tcp1 [8/256/0/180] [ 2568.073484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.51@tcp [ 2568.127625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2568.211102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 2568.250684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 2568.279469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 2568.314695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2568.372263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 2568.408470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.51@tcp [ 2568.435596] LNetError: 193091:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.51@tcp to recovery [ 2568.548464] LNet: There was an unexpected network error while writing to 192.168.203.51: rc = -22 [ 2570.834389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 2574.688523] LNet: There was an unexpected network error while writing to 192.168.203.51: rc = -22 [ 2574.697164] LNet: Skipped 1 previous similar message [ 2579.937849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 2579.955803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 2579.972836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2585.513277] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 11:37:08 (1788190628) [ 2585.615460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2585.631083] LNet: 193801:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2585.651920] LNetError: 193801:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2585.662668] LNetError: 193801:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2585.674094] LNet: Removed LNI 192.168.203.51@tcp [ 2585.702247] LNet: Removed LNI 192.168.203.51@tcp1 [ 2585.727389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2585.817718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2585.872084] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 2586.650791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2586.668896] LNet: Added LNI 192.168.203.51@tcp1 [8/256/0/180] [ 2587.410279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.51@tcp [ 2587.538619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 2587.582712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 2587.661225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 2587.703431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2587.758224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 2587.789471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.51@tcp --health 0 [ 2587.903462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 2588.831766] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.51@tcp [ 2588.832460] LNet: There was an unexpected network error while writing to 192.168.203.51: rc = -22 [ 2588.850835] LNet: Skipped 1 previous similar message [ 2602.367602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 2602.555114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 2602.602445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2602.636983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.51@tcp --health 500 [ 2602.751555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 2608.408519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 2608.516816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2608.540258] LNet: 194284:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2608.547348] LNetError: 194284:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2608.552950] LNetError: 194284:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2608.557777] LNet: Removed LNI 192.168.203.51@tcp [ 2608.568996] LNet: Removed LNI 192.168.203.51@tcp1 [ 2608.584789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2608.636255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2608.660026] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 2609.473695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2609.493780] LNet: Added LNI 192.168.203.51@tcp1 [8/256/0/180] [ 2610.282355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.51@tcp [ 2610.324207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2610.378198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 2610.401887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 2610.448479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 2610.496339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2610.542516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 2610.562513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.51@tcp --health 0 [ 2611.679545] LNet: There was an unexpected network error while writing to 192.168.203.51: rc = -22 [ 2611.690731] LNet: Skipped 3 previous similar messages [ 2611.934609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 2622.175698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 2622.217811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 2622.259814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2628.457258] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 11:37:51 (1788190671) [ 2628.875305] LNet: 195330:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2628.890403] LNetError: 195330:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2628.896848] LNetError: 195330:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2628.912275] LNet: Removed LNI 192.168.203.51@tcp [ 2628.933468] LNet: Removed LNI 192.168.203.51@tcp1 [ 2629.467996] Key type .llcrypt unregistered [ 2629.470242] Key type ._llcrypt unregistered [ 2634.877992] Key type ._llcrypt registered [ 2634.880804] Key type .llcrypt registered [ 2635.098040] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2635.122469] alg: No test for adler32 (adler32-zlib) [ 2636.040275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2636.070904] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 2643.690948] Lustre: DEBUG MARKER: Initial discovery [ 2643.766765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.151@tcp [ 2645.787892] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 2645.849964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 2645.894083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.151@tcp [ 2645.920599] LNet: There was an unexpected network error while writing to 192.168.203.151: rc = -22 [ 2645.948963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 2645.991281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.203.151@tcp -r 1 -m GET -l 3 [ 2646.044134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.203.151@tcp -r 1 -m GET -e local_error [ 2646.083898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.203.151@tcp -r 1 -m PUT -l 6 [ 2646.306851] LNet: 1 local NIs in recovery (showing 1): 192.168.203.51@tcp [ 2647.469257] Lustre: DEBUG MARKER: Force oleg351-client.virtnet to discover 192.168.203.151@tcp (in background) [ 2647.534710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.151@tcp [ 2648.730427] Lustre: DEBUG MARKER: Force oleg351-server to discover 192.168.203.51@tcp [ 2650.597349] LNet: There was an unexpected network error while writing to 192.168.203.151: rc = -22 [ 2650.616331] Lustre: DEBUG MARKER: Wait for 196080 [ 2652.037626] Lustre: DEBUG MARKER: Finished wait on 196080 [ 2652.204280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 2652.246988] LNet: There was an unexpected network error while writing to 192.168.203.151: rc = -22 [ 2652.259607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 2652.459581] LNet: 196537:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2652.470532] LNetError: 196537:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2652.497923] LNet: Removed LNI 192.168.203.51@tcp [ 2652.853680] Key type .llcrypt unregistered [ 2652.856286] Key type ._llcrypt unregistered [ 2666.478958] Key type ._llcrypt registered [ 2666.481433] Key type .llcrypt registered [ 2667.752571] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 11:38:29 (1788190709) [ 2668.375750] Key type .llcrypt unregistered [ 2668.379504] Key type ._llcrypt unregistered [ 2674.523515] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2674.664634] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2674.680368] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2674.689424] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2677.256661] Key type ._llcrypt registered [ 2677.259595] Key type .llcrypt registered [ 2677.385424] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2677.394695] alg: No test for adler32 (adler32-zlib) [ 2678.361860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2678.445765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2678.488516] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 2679.463831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2679.487134] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2687.478212] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 11:38:49 (1788190729) [ 2687.655602] LNet: 198600:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2687.661108] LNetError: 198600:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2687.667399] LNet: Removed LNI 10.1.2.3@tcp [ 2687.712031] LNet: Removed LNI 192.168.203.51@tcp [ 2688.406669] Key type .llcrypt unregistered [ 2688.411071] Key type ._llcrypt unregistered [ 2693.790960] Key type ._llcrypt registered [ 2693.793962] Key type .llcrypt registered [ 2693.939815] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2693.949908] alg: No test for adler32 (adler32-zlib) [ 2694.802965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2694.841320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2694.874113] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 2695.490689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2695.506576] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2696.258432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.203.51@tcp [ 2697.313334] LNetError: Unexpected error -22 connecting to 192.168.203.51@tcp at host 192.168.203.51:988 [ 2701.467074] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 11:39:03 (1788190743) [ 2701.793965] LNet: 199736:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2701.805855] LNetError: 199736:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2701.813479] LNet: Removed LNI 10.1.2.3@tcp [ 2701.837963] LNet: Removed LNI 192.168.203.51@tcp [ 2702.357447] Key type .llcrypt unregistered [ 2702.361213] Key type ._llcrypt unregistered [ 2708.603552] Key type ._llcrypt registered [ 2708.605903] Key type .llcrypt registered [ 2708.934226] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2708.965394] alg: No test for adler32 (adler32-zlib) [ 2709.997450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2710.080517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2710.113040] LNet: Added LNI 192.168.203.51@tcp1 [8/256/0/180] [ 2710.673731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2710.688880] LNet: Added LNI 192.168.203.51@tcp2 [8/256/0/180] [ 2711.492206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.203.51@tcp1 --nid 192.168.203.51@tcp2 [ 2715.656849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.51@tcp1 192.168.203.51@tcp2 [ 2715.672958] LNetError: 200567:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.203.51@tcp1 to 192.168.203.51@tcp2 - Route table may be misconfigured [ 2715.691187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.51@tcp2 192.168.203.51@tcp1 [ 2722.553702] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 11:39:25 (1788190765) [ 2722.876694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2722.906419] LNet: 201140:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2722.918157] LNetError: 201140:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2722.924941] LNet: Removed LNI 192.168.203.51@tcp1 [ 2723.943417] LNet: Removed LNI 192.168.203.51@tcp2 [ 2724.001771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2724.047992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2724.091178] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 2724.655099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2724.682837] LNet: Added LNI 192.168.203.51@tcp1 [8/256/0/180] [ 2725.737452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.51@tcp [ 2725.799585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.51@tcp 192.168.203.51@tcp [ 2725.851675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.51@tcp1 192.168.203.51@tcp1 [ 2725.902094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.51@tcp -d 192.168.203.51@tcp -e network_timeout [ 2725.952867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.51@tcp -d 192.168.203.51@tcp1 -e network_timeout [ 2726.009220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp -e network_timeout [ 2726.055870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp1 -e network_timeout [ 2726.114733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.51@tcp [ 2726.161477] LNet: There was an unexpected network error while writing to 192.168.203.51: rc = -22 [ 2726.179417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 2726.306485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 2726.455946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 2727.137199] LNet: 1 local NIs in recovery (showing 1): 192.168.203.51@tcp [ 2733.047407] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 11:39:35 (1788190775) [ 2733.166541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2733.210536] LNet: 202001:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2733.248918] LNetError: 202001:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2733.259995] LNetError: 202001:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2734.303643] LNet: Removed LNI 192.168.203.51@tcp [ 2734.316480] LNetError: 202001:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2734.345206] LNet: Removed LNI 192.168.203.51@tcp1 [ 2734.394915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2734.529699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 2734.880300] LNet: 202039:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2735.373537] Key type .llcrypt unregistered [ 2735.377987] Key type ._llcrypt unregistered [ 2746.704973] Key type ._llcrypt registered [ 2746.711875] Key type .llcrypt registered [ 2748.502182] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 11:39:50 (1788190790) [ 2748.850384] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2748.869450] alg: No test for adler32 (adler32-zlib) [ 2749.714671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2750.771880] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 2752.217286] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 11:39:54 (1788190794) [ 2752.351934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2752.376289] LNet: 203127:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2752.398052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2752.461707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2752.485972] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 2753.256645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2753.283314] LNet: Added LNI 192.168.203.51@tcp1 [8/256/0/180] [ 2754.207162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.51@tcp [ 2754.290387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.51@tcp1 [ 2754.343984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.51@tcp1 [ 2760.142434] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 11:40:02 (1788190802) [ 2762.494190] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg351-server [ 2764.480299] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 11:40:06 (1788190806) [ 2766.750759] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg351-server [ 2769.003302] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 11:40:10 (1788190810) [ 2770.902916] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg351-server [ 2772.644267] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 11:40:14 (1788190814) [ 2774.556036] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg351-server [ 2776.322410] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 11:40:18 (1788190818) [ 2778.482262] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg351-server [ 2780.411069] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 11:40:22 (1788190822) [ 2782.356433] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg351-server [ 2784.190200] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 11:40:26 (1788190826) [ 2786.162678] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg351-server [ 2788.165810] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 11:40:30 (1788190830) [ 2790.077973] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg351-server [ 2791.846257] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 11:40:34 (1788190834) [ 2793.847638] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg351-server [ 2795.478433] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 11:40:37 (1788190837) [ 2795.569729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2795.608307] LNet: 207004:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2795.620451] LNetError: 207004:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2795.632474] LNet: Removed LNI 192.168.203.51@tcp [ 2796.713350] LNet: Removed LNI 192.168.203.51@tcp1 [ 2796.752018] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2796.853036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2796.888510] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 2797.576533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 2797.647957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.51@tcp [ 2797.836098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2797.859505] LNet: 207159:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2797.869735] LNetError: 207159:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2797.879757] LNetError: 207159:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2797.898685] LNet: Removed LNI 192.168.203.51@tcp [ 2797.935450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2798.016991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 2798.043947] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 2798.681789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.51@tcp [ 2798.847432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2798.870373] LNetError: 207305:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2798.884767] LNet: Removed LNI 192.168.203.51@tcp [ 2798.923190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2798.974438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2798.997115] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 2799.643898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 2799.717877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.51@tcp [ 2799.906697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2799.928347] LNet: 207452:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2799.933891] LNet: 207452:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2800.999873] LNet: Removed LNI 192.168.203.51@tcp [ 2801.030338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2801.072573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 2801.101463] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 2801.729706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.51@tcp [ 2801.915518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2801.943676] LNetError: 207597:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2801.948562] LNetError: 207597:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2802.985790] LNet: Removed LNI 192.168.203.51@tcp [ 2803.018688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2803.047508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2803.063245] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 2803.635652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 2803.741752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.51@tcp [ 2803.949938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2803.967821] LNet: 207744:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2803.972138] LNet: 207744:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2805.030622] LNet: Removed LNI 192.168.203.51@tcp [ 2805.056078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2805.111958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 2805.132068] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 2805.796410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.51@tcp [ 2806.048386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2806.090208] LNetError: 207890:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2806.097400] LNetError: 207890:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2807.143428] LNet: Removed LNI 192.168.203.51@tcp [ 2807.159510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2807.198295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2807.217204] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 2807.922839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 2808.031137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.51@tcp [ 2808.234475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2809.267465] LNet: Removed LNI 192.168.203.51@tcp [ 2809.320276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2809.421979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 2809.471197] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 2810.410889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.51@tcp [ 2810.538289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2811.626103] LNet: Removed LNI 192.168.203.51@tcp [ 2811.677434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2811.720909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2811.737548] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 2812.744347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 2812.883461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.51@tcp [ 2814.525695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2814.585303] LNet: 208339:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2814.605979] LNet: 208339:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2814.618610] LNetError: 208339:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2814.627811] LNetError: 208339:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2815.671458] LNet: Removed LNI 192.168.203.51@tcp [ 2815.739650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2815.910924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 2815.996692] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 2817.179434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.51@tcp [ 2817.586973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2818.691449] LNet: Removed LNI 192.168.203.51@tcp [ 2818.712781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2818.758194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2818.775545] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 2819.785363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 2819.934799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.51@tcp [ 2821.767612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2822.891320] LNet: Removed LNI 192.168.203.51@tcp [ 2822.950934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2823.069610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 2823.125834] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 2824.293419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.51@tcp [ 2824.847609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2825.959589] LNet: Removed LNI 192.168.203.51@tcp [ 2826.014727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2826.144991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2826.191346] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 2827.275494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 2827.417635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2827.445167] LNet: Removed LNI 192.168.203.51@tcp [ 2827.478628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2827.562783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2827.610787] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 2828.826995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 2828.916734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2828.993412] LNet: Removed LNI 192.168.203.51@tcp [ 2829.042386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2829.181709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 2829.366054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2829.422484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2829.563511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 2829.825122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2829.878053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2829.918637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 2829.973460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2830.023915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2830.094343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 2830.132481] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 2831.260989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2831.290572] LNet: 209226:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2831.293366] LNet: 209226:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 2831.326671] LNetError: 209226:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2831.350791] LNetError: 209226:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2831.382154] LNet: Removed LNI 192.168.203.51@tcp [ 2831.424645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2831.537598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2838.685198] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 11:41:20 (1788190880) [ 2838.766324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2838.810328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2838.956353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 2839.014356] LNet: Added LNI 192.168.203.51@tcp231 [8/256/0/180] [ 2840.414306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2840.436512] LNet: Removed LNI 192.168.203.51@tcp231 [ 2840.456185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2840.545486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 2840.571836] LNet: Added LNI 192.168.203.51@tcp231 [8/256/0/180] [ 2840.579856] LNet: 209954:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2840.855585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 2840.884955] LNet: Removed LNI 192.168.203.51@tcp231 [ 2840.894965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 2840.911080] LNet: Added LNI 192.168.203.51@tcp231 [8/256/0/180] [ 2841.937861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2841.976784] LNet: Removed LNI 192.168.203.51@tcp231 [ 2841.998835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2842.093367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 2842.135213] LNet: Added LNI 192.168.203.51@tcp231 [8/256/0/0] [ 2842.140546] LNet: 210106:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2842.781200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 2842.814937] LNet: Removed LNI 192.168.203.51@tcp231 [ 2842.837622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 2842.881183] LNet: Added LNI 192.168.203.51@tcp231 [8/256/0/0] [ 2844.115693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2844.138656] LNet: Removed LNI 192.168.203.51@tcp231 [ 2844.155995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2844.213777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 2844.234620] LNet: Added LNI 192.168.203.51@tcp231 [8/256/0/60] [ 2844.241033] LNet: 210258:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2844.739953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 2844.758216] LNet: Removed LNI 192.168.203.51@tcp231 [ 2844.769121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 2844.787062] LNet: Added LNI 192.168.203.51@tcp231 [8/256/0/60] [ 2845.881369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2845.942123] LNet: Removed LNI 192.168.203.51@tcp231 [ 2845.973744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2846.027427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 2846.054215] LNet: Added LNI 192.168.203.51@tcp231 [8/256/0/180] [ 2846.371040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 2846.398962] LNet: Removed LNI 192.168.203.51@tcp231 [ 2846.407470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 2846.421929] LNet: Added LNI 192.168.203.51@tcp231 [8/256/0/180] [ 2852.235273] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 11:41:34 (1788190894) [ 2852.465175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2852.499884] LNet: Removed LNI 192.168.203.51@tcp231 [ 2852.536907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2852.595186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 2852.653549] LNet: Added LNI 192.168.203.51@tcp232 [8/256/0/180] [ 2853.941769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 2854.037522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 2854.100581] LNet: Removed LNI 192.168.203.51@tcp232 [ 2854.121255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 2854.154701] LNet: Added LNI 192.168.203.51@tcp232 [8/256/0/180] [ 2854.801244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 2854.927366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 2854.985401] LNet: Removed LNI 192.168.203.51@tcp232 [ 2855.018589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 2855.078528] LNet: Added LNI 192.168.203.51@tcp232 [8/256/0/180] [ 2856.376000] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 2856.437834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 2856.485885] LNet: Removed LNI 192.168.203.51@tcp232 [ 2856.504544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 2856.533297] LNet: Added LNI 192.168.203.51@tcp232 [8/256/0/180] [ 2857.511718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 2857.558924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 2857.597633] LNet: Removed LNI 192.168.203.51@tcp232 [ 2857.611246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 2865.922298] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 11:41:47 (1788190907) [ 2866.887838] LNet: 212248:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2866.890749] LNet: 212248:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 6 previous similar messages [ 2867.656320] Key type .llcrypt unregistered [ 2867.667256] Key type ._llcrypt unregistered [ 2875.822713] Key type ._llcrypt registered [ 2875.829759] Key type .llcrypt registered [ 2876.093124] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2876.107360] alg: No test for adler32 (adler32-zlib) [ 2877.165215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2877.209491] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 2885.350025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.151@tcp [ 2893.993512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2894.032741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2894.059293] LNet: Added LNI 192.168.203.51@tcp1 [8/256/0/180] [ 2895.702732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp -m GET -e local_interrupt -r 2 [ 2895.751537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp1 -m GET -e local_interrupt -r 2 [ 2895.788117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp -m GET -e local_interrupt -r 2 [ 2895.822143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp1 -m GET -e local_interrupt -r 2 [ 2895.897114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp -m GET -e local_interrupt -r 2 [ 2895.954973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp1 -m GET -e local_interrupt -r 2 [ 2895.992644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp -m GET -e local_interrupt -r 2 [ 2896.071502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp1 -m GET -e local_interrupt -r 2 [ 2896.131905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.51@tcp 192.168.203.151@tcp [ 2896.161789] LNet: There was an unexpected network error while writing to 192.168.203.151: rc = -22 [ 2896.164915] LNet: 1 local NIs in recovery (showing 1): 192.168.203.51@tcp [ 2896.322981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2896.345795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2896.365256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2896.398762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2896.423817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2896.474725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2896.512871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2896.564897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2896.613941] LNet: 212909:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2896.633640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2896.679909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2896.730521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2896.796068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2896.831281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2896.855325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2897.421273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp -m GET -e local_dropped -r 2 [ 2897.486385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp1 -m GET -e local_dropped -r 2 [ 2897.523802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp -m GET -e local_dropped -r 2 [ 2897.545072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp1 -m GET -e local_dropped -r 2 [ 2897.579209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp -m GET -e local_dropped -r 2 [ 2897.633145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp1 -m GET -e local_dropped -r 2 [ 2897.686535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp -m GET -e local_dropped -r 2 [ 2897.745474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp1 -m GET -e local_dropped -r 2 [ 2897.790731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.51@tcp 192.168.203.151@tcp [ 2897.852211] LNet: There was an unexpected network error while writing to 192.168.203.151: rc = -22 [ 2898.102465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2898.140585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2898.204304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2898.258559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2898.303149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2898.341588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2898.368622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2898.392840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2898.407107] LNet: 213029:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2898.418637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2898.477689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2898.529972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2898.590665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2898.665904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2898.724575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2899.190110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp -m GET -e local_aborted -r 2 [ 2899.220128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp1 -m GET -e local_aborted -r 2 [ 2899.255983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp -m GET -e local_aborted -r 2 [ 2899.288716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp1 -m GET -e local_aborted -r 2 [ 2899.317198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp -m GET -e local_aborted -r 2 [ 2899.344713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp1 -m GET -e local_aborted -r 2 [ 2899.396900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp -m GET -e local_aborted -r 2 [ 2899.431546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp1 -m GET -e local_aborted -r 2 [ 2899.502875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.51@tcp 192.168.203.151@tcp [ 2899.544226] LNet: There was an unexpected network error while writing to 192.168.203.151: rc = -22 [ 2899.728971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2899.755941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2899.778589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2899.800094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2899.828539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2899.875148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2899.928498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2899.957557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2899.997166] LNet: 213150:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2900.024679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2900.116091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2900.161612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2900.219696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2900.276234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2900.333698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2901.175212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp -m GET -e local_no_route -r 2 [ 2901.238157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp1 -m GET -e local_no_route -r 2 [ 2901.296412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp -m GET -e local_no_route -r 2 [ 2901.337588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp1 -m GET -e local_no_route -r 2 [ 2901.383715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp -m GET -e local_no_route -r 2 [ 2901.451441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp1 -m GET -e local_no_route -r 2 [ 2901.536804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp -m GET -e local_no_route -r 2 [ 2901.602246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp1 -m GET -e local_no_route -r 2 [ 2901.691151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.51@tcp 192.168.203.151@tcp [ 2901.739215] LNet: There was an unexpected network error while writing to 192.168.203.151: rc = -22 [ 2902.049238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2902.100260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2902.149162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2902.194992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2902.230962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2902.271815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2902.312534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2902.363386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2902.380717] LNet: 213270:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2902.420914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2902.468585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2902.517108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2902.571626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2902.599270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2902.626765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2903.174424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp -m GET -e local_timeout -r 2 [ 2903.207402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp1 -m GET -e local_timeout -r 2 [ 2903.262048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp -m GET -e local_timeout -r 2 [ 2903.306140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp1 -m GET -e local_timeout -r 2 [ 2903.353597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp -m GET -e local_timeout -r 2 [ 2903.405746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp1 -m GET -e local_timeout -r 2 [ 2903.455226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp -m GET -e local_timeout -r 2 [ 2903.494750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp1 -m GET -e local_timeout -r 2 [ 2903.543967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.51@tcp 192.168.203.151@tcp [ 2903.826909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2903.866398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2903.904842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2903.954430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2904.002116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2904.029972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2904.069038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2904.117948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2904.152311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2904.179383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2904.215115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2904.250374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2904.281900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2904.314697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2904.742333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp -m GET -e remote_dropped -r 2 [ 2904.805177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp1 -m GET -e remote_dropped -r 2 [ 2904.858209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp -m GET -e remote_dropped -r 2 [ 2904.900124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp1 -m GET -e remote_dropped -r 2 [ 2904.937816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp -m GET -e remote_dropped -r 2 [ 2904.976069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp1 -m GET -e remote_dropped -r 2 [ 2905.019470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp -m GET -e remote_dropped -r 2 [ 2905.082120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp1 -m GET -e remote_dropped -r 2 [ 2905.118930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.51@tcp 192.168.203.151@tcp [ 2905.141544] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.151@tcp [ 2905.315359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2905.346560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2905.383843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2905.406936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2905.446110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2905.507247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2905.565897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2905.597211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2905.651184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2905.692860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2905.724714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2905.757695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2905.801776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2905.840539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2911.715657] LNet: 213585:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2911.724206] LNetError: 213585:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2911.733221] LNet: Removed LNI 192.168.203.51@tcp [ 2911.741255] LNet: Removed LNI 192.168.203.51@tcp1 [ 2912.180536] Key type .llcrypt unregistered [ 2912.184038] Key type ._llcrypt unregistered [ 2928.097397] Key type ._llcrypt registered [ 2928.103540] Key type .llcrypt registered [ 2929.361975] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 11:42:51 (1788190971) [ 2929.896813] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2929.905525] alg: No test for adler32 (adler32-zlib) [ 2930.789211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2930.834182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2930.882417] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 2931.902670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 2931.986400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.51@tcp [ 2932.020600] LNetError: 214561:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.51@tcp to recovery [ 2932.154250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.203.51@tcp [ 2932.303806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2941.495079] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 11:43:03 (1788190983) [ 2941.683974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2941.772295] LNet: 215146:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2941.785624] LNetError: 215146:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2941.820078] LNet: Removed LNI 192.168.203.51@tcp [ 2941.869192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2941.991880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2942.030541] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 2943.170871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2943.212580] LNet: Added LNI 192.168.203.51@tcp1 [8/256/0/180] [ 2944.941763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.203.51@tcp [ 2944.989591] LNetError: 215417:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.51@tcp to recovery [ 2945.060533] LNet: 1 local NIs in recovery (showing 1): 192.168.203.51@tcp [ 2945.071919] LNet: There was an unexpected network error while writing to 192.168.203.51: rc = -22 [ 2945.246845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.203.51@tcp [ 2955.173829] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 11:43:17 (1788190997) [ 2956.929612] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg351-server [ 2958.960172] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 11:43:21 (1788191001) [ 2960.798493] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg351-server [ 2962.742801] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 11:43:24 (1788191004) [ 2964.539904] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg351-server [ 2966.243217] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 11:43:28 (1788191008) [ 2966.490857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2966.513738] LNet: 217041:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2966.521537] LNetError: 217041:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2966.526870] LNet: Removed LNI 192.168.203.51@tcp [ 2966.541673] LNet: Removed LNI 192.168.203.51@tcp1 [ 2966.563367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2966.653655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2966.688991] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 2967.348836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 2967.446321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 2967.527234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2967.692434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 2967.747467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2967.871815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 2974.111628] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 11:43:36 (1788191016) [ 2974.214710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2974.244383] LNet: 217767:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2974.249409] LNetError: 217767:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2974.255867] LNetError: 217767:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2974.266864] LNet: Removed LNI 192.168.203.51@tcp [ 2974.310399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2974.375628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2974.421288] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 2981.681875] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 11:43:43 (1788191023) [ 2982.085366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2982.104278] LNet: 218476:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2982.114879] LNetError: 218476:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2982.131907] LNet: Removed LNI 192.168.203.51@tcp [ 2982.188830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2982.262975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2982.284330] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 2988.887297] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 11:43:51 (1788191031) [ 2990.184711] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 2991.935392] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 11:43:54 (1788191034) [ 2992.431482] LNet: 219512:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2992.443944] LNetError: 219512:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2992.460886] LNet: Removed LNI 192.168.203.51@tcp [ 2993.124861] Key type .llcrypt unregistered [ 2993.126572] Key type ._llcrypt unregistered [ 2999.363271] Key type ._llcrypt registered [ 2999.366311] Key type .llcrypt registered [ 2999.574852] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2999.598547] alg: No test for adler32 (adler32-zlib) [ 3000.531579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 3000.560596] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 3008.347951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.151@tcp [ 3012.623059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.203.151@tcp [ 3012.910844] LNet: 219927:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3012.919524] LNetError: 219927:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3012.936195] LNet: Removed LNI 192.168.203.51@tcp [ 3013.286407] Key type .llcrypt unregistered [ 3013.288923] Key type ._llcrypt unregistered [ 3024.147442] Key type ._llcrypt registered [ 3024.149071] Key type .llcrypt registered [ 3025.368302] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 3026.958648] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 3028.546474] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 11:44:30 (1788191070) [ 3029.821665] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 3031.200411] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 11:44:33 (1788191073) [ 3032.930365] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg351-server [ 3034.645873] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 11:44:36 (1788191076) [ 3036.662329] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg351-server [ 3038.594148] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 11:44:40 (1788191080) [ 3040.050425] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3040.083642] alg: No test for adler32 (adler32-zlib) [ 3041.209343] Lustre: Lustre: Build Version: 2.17.57_85_gb71fc30 [ 3041.551815] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 3043.279230] Key type lgssc registered [ 3044.398458] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3064.284787] Key type lgssc unregistered [ 3064.563124] LNet: 223098:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3064.572682] LNetError: 223098:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3064.587641] LNet: Removed LNI 192.168.203.51@tcp [ 3065.427497] Key type .llcrypt unregistered [ 3065.430463] Key type ._llcrypt unregistered [ 3086.153884] Key type ._llcrypt registered [ 3086.157265] Key type .llcrypt registered [ 3087.674155] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 11:45:29 (1788191129) [ 3088.082255] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3088.104586] alg: No test for adler32 (adler32-zlib) [ 3089.015122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3089.061644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 3089.103531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 3089.160568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 3089.211602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 3096.086144] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 11:45:38 (1788191138) [ 3096.192776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3096.260576] LNet: 224504:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3096.294330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3096.381654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3096.416457] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 3097.525195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3097.556983] LNet: 224654:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3097.571368] LNetError: 224654:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3097.599258] LNet: Removed LNI 192.168.203.51@tcp [ 3097.641693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3097.740468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 3097.775594] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 3098.569841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3098.602254] LNet: 224682:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3098.609069] LNetError: 224682:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3098.622965] LNet: Removed LNI 192.168.203.51@tcp [ 3098.636696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3098.714538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 3098.744961] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 3107.017529] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 11:45:48 (1788191148) [ 3107.273124] LNet: 225272:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3107.278668] LNetError: 225272:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3107.303265] LNet: Removed LNI 192.168.203.51@tcp [ 3108.301529] Key type .llcrypt unregistered [ 3108.304164] Key type ._llcrypt unregistered [ 3116.084995] Key type ._llcrypt registered [ 3116.087461] Key type .llcrypt registered [ 3116.286651] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 3116.323866] alg: No test for adler32 (adler32-zlib) [ 3117.248329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3117.364182] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 3117.377052] LNet: *** cfs_fail_loc=e003, val=100*** [ 3118.105361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.51@tcp [ 3118.126456] LNet: *** cfs_fail_loc=e003, val=100*** [ 3118.129671] LNet: Skipped 2 previous similar messages [ 3118.135381] Ping reply received with mapping: [ 3118.138776] found metadata reply NI[192.168.203.51@tcp] [ 3118.141932] 33 cb a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 3118.144957] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 3118.148143] 5a 5a 5a 5a 5a 5a 5a 5a [ 3118.233545] LNet: 225667:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3118.238156] LNetError: 225667:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3118.249995] LNet: Removed LNI 192.168.203.51@tcp [ 3118.822477] Key type .llcrypt unregistered [ 3118.826603] Key type ._llcrypt unregistered [ 3129.641641] Key type ._llcrypt registered [ 3129.656364] Key type .llcrypt registered [ 3131.420933] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 11:46:12 (1788191172) [ 3131.965499] Key type .llcrypt unregistered [ 3131.967225] Key type ._llcrypt unregistered [ 3138.859701] Key type ._llcrypt registered [ 3138.868895] Key type .llcrypt registered [ 3139.065827] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 3139.089510] alg: No test for adler32 (adler32-zlib) [ 3140.099616] LNetError: 226702:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 3140.815976] Key type .llcrypt unregistered [ 3140.817642] Key type ._llcrypt unregistered [ 3146.220618] Key type ._llcrypt registered [ 3146.223491] Key type .llcrypt registered [ 3147.912338] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 11:46:29 (1788191189) [ 3148.689286] Key type .llcrypt unregistered [ 3148.691069] Key type ._llcrypt unregistered [ 3156.063949] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 3156.068628] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 3158.548313] Key type ._llcrypt registered [ 3158.551537] Key type .llcrypt registered [ 3158.721855] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 3158.735399] alg: No test for adler32 (adler32-zlib) [ 3159.780618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3159.894280] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 3170.112542] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 11:46:52 (1788191212) [ 3170.275332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3170.329611] LNet: 228473:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3170.346151] LNetError: 228473:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 3170.401644] LNet: Removed LNI 10.1.2.3@tcp [ 3170.417490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3170.953411] Key type .llcrypt unregistered [ 3170.954929] Key type ._llcrypt unregistered [ 3176.377683] Key type ._llcrypt registered [ 3176.379083] Key type .llcrypt registered [ 3176.472707] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 3176.487876] alg: No test for adler32 (adler32-zlib) [ 3177.406482] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 3177.486513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.51@tcp [ 3177.653874] LNet: 228762:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3177.666953] LNetError: 228762:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3177.697938] LNet: Removed LNI 192.168.203.51@tcp [ 3178.403544] Key type .llcrypt unregistered [ 3178.408572] Key type ._llcrypt unregistered [ 3187.969633] Key type ._llcrypt registered [ 3187.971929] Key type .llcrypt registered [ 3189.253135] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 11:47:11 (1788191231) [ 3189.797490] Key type .llcrypt unregistered [ 3189.799277] Key type ._llcrypt unregistered [ 3194.636593] Key type ._llcrypt registered [ 3194.641864] Key type .llcrypt registered [ 3194.928184] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 3194.945690] alg: No test for adler32 (adler32-zlib) [ 3202.353496] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 11:47:24 (1788191244) [ 3202.658282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3202.701475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3202.808908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3202.840865] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 3203.772523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3203.825095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 3203.876149] LNet: Removed LNI 192.168.203.51@tcp [ 3204.222820] LNet: 230501:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3204.582545] Key type .llcrypt unregistered [ 3204.585666] Key type ._llcrypt unregistered [ 3216.121668] Key type ._llcrypt registered [ 3216.127472] Key type .llcrypt registered [ 3217.466983] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 11:47:39 (1788191259) [ 3217.843534] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3217.853478] alg: No test for adler32 (adler32-zlib) [ 3218.743830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3218.808141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3218.855399] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 3219.641666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.51@tcp [ 3219.715579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.203.51@tcp [ 3219.757668] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 3219.763694] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 3219.772091] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 3219, 0, 0 [ 3219.786573] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 3219.793328] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 3219.801250] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 3219.806371] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 3219, 0, 0 [ 3219.816179] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 3219.819155] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 3219.822338] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 3219.825702] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 3219, 0, 3269 [ 3219.830329] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 3219.833381] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 3219.839178] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 3219.842665] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 3219, 0, 3269 [ 3219.848339] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 3226.425944] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 11:47:48 (1788191268) [ 3227.098758] LNet: 232037:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3227.110947] LNetError: 232037:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3227.151527] LNet: Removed LNI 192.168.203.51@tcp [ 3228.024976] Key type .llcrypt unregistered [ 3228.028840] Key type ._llcrypt unregistered [ 3234.575594] Key type ._llcrypt registered [ 3234.577109] Key type .llcrypt registered [ 3234.819765] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 3234.846757] alg: No test for adler32 (adler32-zlib) [ 3235.786572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 3235.829136] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 3243.637962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.151@tcp [ 3252.644936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3252.708570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 3252.739466] LNet: Added LNI 192.168.203.51@tcp1 [8/256/0/180] [ 3253.943178] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 3254.146738] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 3254.149222] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 3254.152068] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 3256.660076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 3256.715681] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 3265.675586] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 11:48:27 (1788191307) [ 3265.834129] LNet: 233534:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3265.843316] LNetError: 233534:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3265.862085] LNet: Removed LNI 192.168.203.51@tcp [ 3265.864823] LNet: Removed LNI 192.168.203.51@tcp1 [ 3265.897529] LNet: Removed LNI 10.1.2.3@tcp99 [ 3266.551341] Key type .llcrypt unregistered [ 3266.556729] Key type ._llcrypt unregistered [ 3273.200958] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 3273.210853] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 3275.857595] Key type ._llcrypt registered [ 3275.859495] Key type .llcrypt registered [ 3276.069567] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 3276.108894] alg: No test for adler32 (adler32-zlib) [ 3276.997235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3277.072481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3277.134792] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 3278.067201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 3278.104717] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 3280.656727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 3280.743352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 3280.866937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 3280.924453] LNetError: 234281:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 3280.994706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 3281.068476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.51@tcp [ 3281.662084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 3281.759917] LNet: 234301:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3281.767926] LNetError: 234301:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 3281.775357] LNet: Removed LNI 10.1.2.3@tcp [ 3282.790737] LNet: Removed LNI 192.168.203.51@tcp [ 3283.601687] Key type .llcrypt unregistered [ 3283.605553] Key type ._llcrypt unregistered [ 3297.188996] Key type ._llcrypt registered [ 3297.191184] Key type .llcrypt registered [ 3298.750300] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 11:49:00 (1788191340) [ 3299.167496] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3299.183057] alg: No test for adler32 (adler32-zlib) [ 3300.054465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3300.091234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3300.106045] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 3300.893584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.51@tcp [ 3300.981639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg351-client@tcp [ 3307.719803] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 11:49:09 (1788191349) [ 3308.158570] LNet: 235832:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3308.163486] LNetError: 235832:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3308.182883] LNet: Removed LNI 192.168.203.51@tcp [ 3308.867411] Key type .llcrypt unregistered [ 3308.869673] Key type ._llcrypt unregistered [ 3314.439170] Key type ._llcrypt registered [ 3314.441173] Key type .llcrypt registered [ 3314.627265] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3314.654490] alg: No test for adler32 (adler32-zlib) [ 3315.619693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 3315.658719] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 3315.714738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 3323.229529] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 11:49:24 (1788191364) [ 3324.927384] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 3326.831452] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 11:49:28 (1788191368) [ 3328.294515] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 3330.173217] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 11:49:32 (1788191372) [ 3330.429027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3330.486378] LNet: 237251:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3330.512492] LNetError: 237251:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3330.531747] LNet: Removed LNI 192.168.203.51@tcp [ 3330.548510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3330.614843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3330.647641] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 3331.696763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 3331.777972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 3340.381732] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 11:49:42 (1788191382) [ 3340.755791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3340.791509] LNet: 237963:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3340.799092] LNetError: 237963:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3340.830053] LNet: Removed LNI 192.168.203.51@tcp [ 3340.895169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3341.026244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 3341.083280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 3342.093360] Key type .llcrypt unregistered [ 3342.107825] Key type ._llcrypt unregistered [ 3355.403674] Key type ._llcrypt registered [ 3355.405795] Key type .llcrypt registered [ 3356.446808] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 11:49:58 (1788191398) [ 3357.040579] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3357.076537] alg: No test for adler32 (adler32-zlib) [ 3358.143605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3358.177862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3358.192923] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 3358.710143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 3358.769480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.51@tcp [ 3364.837289] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 11:50:07 (1788191407) [ 3365.035177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3365.063534] LNet: 239524:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3365.080506] LNetError: 239524:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3365.099949] LNet: Removed LNI 192.168.203.51@tcp [ 3365.156876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3365.221753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 3365.253535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 3370.979676] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 11:50:13 (1788191413) [ 3371.243626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3371.274791] LNet: 240105:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3371.295188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3371.376760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 3371.651087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 3371.677293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 3371.740185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 3372.086126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 3372.146477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 3372.283840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3372.314912] LNet: 240138:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3372.385541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3372.447774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 3372.892907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 3372.922731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 3372.979456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 3373.520052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 3373.558720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 3380.851193] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 11:50:22 (1788191422) [ 3381.115248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3381.157310] LNet: 240738:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3381.188391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3381.312990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3381.330182] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 3382.133584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 3382.162967] LNet: Added LNI 192.168.203.51@tcp2 [8/256/0/180] [ 3383.132694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 3383.221679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.51@tcp [ 3383.601771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3383.638798] LNetError: 241018:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3383.650983] LNet: Removed LNI 192.168.203.51@tcp [ 3383.694320] LNet: Removed LNI 192.168.203.51@tcp2 [ 3383.767295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3383.857897] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 3383.876867] LNet: Added LNI 192.168.203.51@tcp2 [8/256/0/180] [ 3383.883463] LNet: 241024:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3384.274945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.51@tcp [ 3384.398365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3385.439147] LNet: Removed LNI 192.168.203.51@tcp [ 3385.444125] LNetError: 241061:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3385.462702] LNetError: 241061:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 3385.491527] LNet: Removed LNI 192.168.203.51@tcp2 [ 3385.552914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3385.659799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3385.701542] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 3386.665166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 3386.709904] LNet: Added LNI 192.168.203.51@tcp2 [8/256/0/180] [ 3387.608526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.51@tcp [ 3387.663564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 3387.786586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.51@tcp [ 3388.059433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3388.083194] LNet: 241346:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3388.086874] LNet: 241346:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 3388.092487] LNetError: 241346:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3389.151372] LNet: Removed LNI 192.168.203.51@tcp [ 3390.182508] LNet: Removed LNI 192.168.203.51@tcp2 [ 3390.215303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3390.310896] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 3390.317955] LNet: Added LNI 192.168.203.51@tcp2 [8/256/0/180] [ 3390.322742] LNet: 241352:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3391.007479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.51@tcp [ 3391.200023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3391.221347] LNet: Removed LNI 192.168.203.51@tcp [ 3392.293583] LNet: Removed LNI 192.168.203.51@tcp2 [ 3392.349442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3392.455410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3392.488131] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 3393.221039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 3393.237493] LNet: Added LNI 192.168.203.51@tcp2 [8/256/0/180] [ 3394.125871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 3394.213270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.51@tcp [ 3394.467741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3394.501033] LNetError: 241669:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3394.510073] LNetError: 241669:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 3394.523590] LNet: Removed LNI 192.168.203.51@tcp [ 3394.543922] LNet: Removed LNI 192.168.203.51@tcp2 [ 3394.591874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3394.713224] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 3394.723427] LNet: Added LNI 192.168.203.51@tcp2 [8/256/0/180] [ 3394.728711] LNet: 241675:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3395.141515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.51@tcp [ 3395.353834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3396.383134] LNet: Removed LNI 192.168.203.51@tcp [ 3396.395882] LNet: Removed LNI 192.168.203.51@tcp2 [ 3396.439365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3396.507523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3396.536908] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 3397.261499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 3397.284499] LNet: Added LNI 192.168.203.51@tcp2 [8/256/0/180] [ 3398.154988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.51@tcp [ 3398.211859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 3398.272562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.51@tcp [ 3398.493286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3398.513927] LNet: 241996:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3398.521251] LNet: 241996:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 3399.584468] LNet: Removed LNI 192.168.203.51@tcp [ 3400.613653] LNet: Removed LNI 192.168.203.51@tcp2 [ 3400.659845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3400.743073] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 3400.757447] LNet: Added LNI 192.168.203.51@tcp2 [8/256/0/180] [ 3400.767301] LNet: 242003:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3401.471073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.51@tcp [ 3401.724896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3401.758839] LNet: Removed LNI 192.168.203.51@tcp [ 3402.798329] LNet: Removed LNI 192.168.203.51@tcp2 [ 3402.830740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3402.928386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3402.945605] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 3403.533633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 3403.553598] LNet: Added LNI 192.168.203.51@tcp2 [8/256/0/180] [ 3404.513587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 3404.575990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 3404.699374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.51@tcp [ 3405.208034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3405.244526] LNetError: 242336:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3405.252749] LNetError: 242336:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 3405.261066] LNet: Removed LNI 192.168.203.51@tcp [ 3405.269851] LNet: Removed LNI 192.168.203.51@tcp2 [ 3405.296298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3405.378342] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 3405.392794] LNet: Added LNI 192.168.203.51@tcp2 [8/256/0/180] [ 3405.401680] LNet: 242342:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3406.236799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.51@tcp [ 3406.614098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3407.649327] LNet: Removed LNI 192.168.203.51@tcp [ 3407.666627] LNet: Removed LNI 192.168.203.51@tcp2 [ 3407.726992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3407.792634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3407.836971] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 3408.495634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 3408.537497] LNet: Added LNI 192.168.203.51@tcp2 [8/256/0/180] [ 3409.598907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.51@tcp [ 3409.660875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 3409.716288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 3409.797139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.51@tcp [ 3410.299613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3411.359326] LNet: Removed LNI 192.168.203.51@tcp [ 3412.459551] LNet: Removed LNI 192.168.203.51@tcp2 [ 3412.475346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3412.541790] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 3412.556339] LNet: Added LNI 192.168.203.51@tcp2 [8/256/0/180] [ 3412.988697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.51@tcp [ 3421.466170] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 3424.201677] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 11:51:05 (1788191465) [ 3424.383569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3424.441238] LNet: 243456:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3424.460182] LNet: 243456:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 3424.474947] LNetError: 243456:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3424.486496] LNetError: 243456:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 3424.497865] LNet: Removed LNI 192.168.203.51@tcp [ 3425.511759] LNet: Removed LNI 192.168.203.51@tcp2 [ 3425.549935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3426.148967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3426.221600] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 3427.166446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 3427.201543] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 3428.346470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3428.368931] LNet: Removed LNI 10.1.2.3@tcp [ 3428.381557] LNet: Removed LNI 192.168.203.51@tcp [ 3428.400870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3428.437885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 3428.476653] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 3428.481097] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 3429.186930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 3429.224865] LNet: Removed LNI 192.168.203.51@tcp [ 3429.253432] LNet: Removed LNI 10.1.2.3@tcp [ 3438.091363] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 11:51:19 (1788191479) [ 3438.241476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3438.347583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3438.475244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3438.511480] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 3439.615792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 3439.687231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.51@tcp [ 3439.766238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.51@tcp [ 3439.837274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.51@tcp [ 3439.929392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.51@tcp [ 3440.022841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.51@tcp [ 3440.141806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.51@tcp [ 3440.174565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.51@tcp [ 3440.237762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.51@tcp [ 3440.289736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.51@tcp [ 3440.412756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.51@tcp [ 3440.687154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 3448.878798] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 11:51:30 (1788191490) [ 3449.043318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3450.086198] LNet: Removed LNI 192.168.203.51@tcp [ 3450.144493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3450.313505] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 3450.520943] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 3450.527474] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 3450.539091] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 3452.972602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3453.010553] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 3453.770963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 3453.804822] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 3456.086391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.51@tcp [ 3456.162596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 3456.255571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.203.51@tcp -m PUT -e local_timeout -r 1 [ 3456.348055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 3456.451833] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 3456.459103] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 3456.708108] LNetError: 245140:0:(lib-move.c:4801:lnet_parse()) 192.168.203.51@tcp, src 192.168.203.51@tcp: Dropping PUT (error -108 looking up sender) [ 3457.759948] LNet: 245600:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3457.776496] LNet: 245600:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 3457.791112] LNetError: 245600:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 3457.804629] LNetError: 245600:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 3457.818555] LNet: Removed LNI 10.1.2.3@tcp [ 3458.866173] LNet: Removed LNI 192.168.203.51@tcp [ 3459.301626] Key type .llcrypt unregistered [ 3459.303726] Key type ._llcrypt unregistered [ 3471.766753] Key type ._llcrypt registered [ 3471.768712] Key type .llcrypt registered [ 3473.091293] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 11:51:55 (1788191515) [ 3473.511315] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3473.527787] alg: No test for adler32 (adler32-zlib) [ 3474.527891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3474.947395] LNet: 246449:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3475.658507] Key type .llcrypt unregistered [ 3475.664919] Key type ._llcrypt unregistered [ 3483.441110] Key type ._llcrypt registered [ 3483.444408] Key type .llcrypt registered [ 3483.773813] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 3483.803365] alg: No test for adler32 (adler32-zlib) [ 3484.926611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 3484.957334] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 3494.051061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.151@tcp [ 3494.312128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 3495.279273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.151@tcp [ 3495.707201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 3495.777188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 3495.831423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 3495.918585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 3495.991762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 3496.054904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 3496.118579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 3496.178059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 3496.206706] LNet: 246896:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3496.229313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 3496.263963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 3496.289263] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.151@tcp [ 3496.312967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 3496.359143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 3496.388876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 3496.422714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 3497.184552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 3497.593908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 3497.634935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 3497.689820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 3497.731989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 3497.814934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 3497.870882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 3497.926570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 3497.970889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 3498.017985] LNet: 247014:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3498.047311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 3498.102927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 3498.173893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 3498.254877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 3498.305836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 3498.402289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 3499.306867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.151@tcp [ 3499.572483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 3499.613961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 3499.641363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 3499.673175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 3499.715316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 3499.758630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 3499.812953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 3499.852795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 3499.877933] LNet: 247131:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3499.911080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 3499.972873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 3500.047695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 3500.089035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 3500.140223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 3500.200956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 3501.049629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 3501.329642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 3501.356757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 3501.386553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 3501.414852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 3501.451304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 3501.483320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 3501.544503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 3501.601882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 3501.673131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 3501.704792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 3501.736934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 3501.761398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 3501.794503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 3501.826303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 3502.172915] LNet: 247294:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3502.180391] LNetError: 247294:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3502.197926] LNet: Removed LNI 192.168.203.51@tcp [ 3502.664387] Key type .llcrypt unregistered [ 3502.666332] Key type ._llcrypt unregistered [ 3519.868532] Key type ._llcrypt registered [ 3519.871676] Key type .llcrypt registered [ 3521.111167] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 11:52:43 (1788191563) [ 3521.465746] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3521.480668] alg: No test for adler32 (adler32-zlib) [ 3522.392559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3522.753729] LNet: 248177:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3523.395371] Key type .llcrypt unregistered [ 3523.404162] Key type ._llcrypt unregistered [ 3529.606509] Key type ._llcrypt registered [ 3529.610537] Key type .llcrypt registered [ 3529.761140] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 3529.772721] alg: No test for adler32 (adler32-zlib) [ 3530.654947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 3530.679544] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 3536.929803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.151@tcp [ 3543.912918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3543.965856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 3543.994065] LNet: Added LNI 192.168.203.51@tcp1 [8/256/0/180] [ 3544.704203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 3545.250843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.151@tcp [ 3545.425348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 3545.450283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 3545.476455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 3545.502466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 3545.523799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 3545.548140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 3545.584897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 3545.610831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 3545.621527] LNet: 248826:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3545.632219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 3545.651565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 3545.693263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 3545.725553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 3545.755299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 3545.802242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 3546.574306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 3546.767177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 3546.793618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 3546.816061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 3546.861919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 3546.888517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 3546.922285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 3546.982952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 3547.046664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 3547.086556] LNet: 248943:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3547.103903] LNet: 2 peer NIs in recovery (showing 2): 192.168.203.151@tcp, 192.168.203.151@tcp1 [ 3547.123723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 3547.176957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 3547.234508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 3547.271661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 3547.318493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 3547.376289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 3548.093475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.151@tcp [ 3548.339862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 3548.376661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 3548.404665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 3548.428135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 3548.455663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 3548.481478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 3548.504978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 3548.534512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 3548.550638] LNet: 249060:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3548.582582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 3548.614697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 3548.641607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 3548.664426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 3548.687877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 3548.721796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 3549.304824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 3549.559428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 3549.588445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 3549.613609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 3549.662923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 3549.692287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 3549.728867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 3549.781858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 3549.844641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 3549.885895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 3549.931991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 3549.965980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 3550.009552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 3550.064277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 3550.121634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 3555.472703] LNet: 249258:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3555.480234] LNetError: 249258:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3555.491636] LNet: Removed LNI 192.168.203.51@tcp [ 3555.503312] LNet: Removed LNI 192.168.203.51@tcp1 [ 3555.910251] Key type .llcrypt unregistered [ 3555.925493] Key type ._llcrypt unregistered [ 3571.757572] Key type ._llcrypt registered [ 3571.761779] Key type .llcrypt registered [ 3573.200130] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 11:53:35 (1788191615) [ 3573.458607] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3573.472035] alg: No test for adler32 (adler32-zlib) [ 3574.347118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3574.425520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3574.446714] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 3574.903722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 3574.916588] LNet: Added LNI 192.168.203.51@tcp2 [8/256/0/180] [ 3575.541415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.51@tcp [ 3575.604899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 3576.671390] LNet: Removed LNI 192.168.203.51@tcp [ 3582.911721] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 11:53:44 (1788191624) [ 3584.502077] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 3586.932933] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 11:53:48 (1788191628) [ 3597.831474] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 11:53:59 (1788191639) [ 3607.730511] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 11:54:09 (1788191649) [ 3615.534207] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 11:54:17 (1788191657) [ 3626.189374] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 11:54:27 (1788191667) [ 3634.973492] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 11:54:36 (1788191676) [ 3644.296315] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 11:54:46 (1788191686) [ 3652.400265] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 11:54:54 (1788191694) [ 3661.031364] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 11:55:03 (1788191703) [ 3668.388582] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 11:55:10 (1788191710) [ 3675.955907] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 11:55:18 (1788191718) [ 3682.889495] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 11:55:25 (1788191725) [ 3689.061699] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 11:55:31 (1788191731) [ 3695.318518] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 11:55:37 (1788191737) [ 3701.770338] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 11:55:44 (1788191744) [ 3709.907417] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 11:55:52 (1788191752) [ 3720.226916] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 11:56:01 (1788191761) [ 3729.617809] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 11:56:11 (1788191771) [ 3729.797836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3729.821319] LNet: 260955:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3729.828623] LNetError: 260955:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3729.846065] LNet: Removed LNI 192.168.203.51@tcp2 [ 3729.869691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3729.947619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3729.967037] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 3730.806221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3730.844659] LNet: 261097:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3730.862229] LNetError: 261097:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3730.883488] LNet: Removed LNI 192.168.203.51@tcp [ 3730.928305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3731.029125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 3731.068290] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 3738.294808] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 11:56:20 (1788191780) [ 3745.335584] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 11:56:27 (1788191787) [ 3752.250232] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 11:56:34 (1788191794) [ 3759.910630] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 11:56:42 (1788191802) [ 3766.486814] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 11:56:48 (1788191808) [ 3773.618902] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 11:56:55 (1788191815) [ 3780.435189] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 11:57:02 (1788191822) [ 3788.001401] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 11:57:09 (1788191829) [ 3794.014550] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 11:57:16 (1788191836) [ 3800.720138] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 11:57:22 (1788191842) [ 3808.131733] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 11:57:30 (1788191850) [ 3815.280211] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 11:57:37 (1788191857) [ 3822.882900] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 3552 sec ========== 11:57:44 (1788191864) [ 3823.178911] LNet: 268562:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3823.186863] LNetError: 268562:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3823.201811] LNet: Removed LNI 192.168.203.51@tcp [ 3823.746248] Key type .llcrypt unregistered [ 3823.751291] Key type ._llcrypt unregistered [ 3837.388442] Key type ._llcrypt registered [ 3837.394349] Key type .llcrypt registered [ 3837.650053] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 3837.668273] alg: No test for adler32 (adler32-zlib) [ 3838.879918] Lustre: Lustre: Build Version: 2.17.57_85_gb71fc30 [ 3839.289157] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 3841.023730] Key type lgssc registered [ 3842.521177] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3906.436875] Lustre: Mounted lustre-client - version 2.17.57_85_gb71fc30 [ 3911.486709] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3918.781560] Lustre: Unmounted lustre-client [ 3937.677856] Key type lgssc unregistered [ 3937.965840] LNet: 271521:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3937.981132] LNetError: 271521:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3938.014950] LNet: Removed LNI 192.168.203.51@tcp [ 3938.901288] Key type .llcrypt unregistered [ 3938.909174] Key type ._llcrypt unregistered