[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 425645775 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895288K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001008] APIC: Switch to symmetric I/O mode setup [ 0.002407] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004014] kvm-guest: setup PV IPIs [ 0.007553] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010006] pid_max: default: 32768 minimum: 301 [ 0.011121] LSM: Security Framework initializing [ 0.012069] Yama: becoming mindful. [ 0.013051] SELinux: Initializing. [ 0.014062] *** VALIDATE selinux *** [ 0.022195] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027374] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028155] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029111] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030104] *** VALIDATE tmpfs *** [ 0.032174] *** VALIDATE proc *** [ 0.033236] *** VALIDATE cgroup *** [ 0.034010] *** VALIDATE cgroup2 *** [ 0.036230] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037157] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039030] Spectre V2 : User space: Vulnerable [ 0.040009] Speculative Store Bypass: Vulnerable [ 0.043155] debug: unmapping init [mem 0xffffffffb5459000-0xffffffffb5460fff] [ 0.045949] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046633] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047025] ... version: 2 [ 0.048012] ... bit width: 48 [ 0.049011] ... generic registers: 4 [ 0.050011] ... value mask: 0000ffffffffffff [ 0.051010] ... max period: 00007fffffffffff [ 0.052013] ... fixed-purpose events: 3 [ 0.053024] ... event mask: 000000070000000f [ 0.055217] rcu: Hierarchical SRCU implementation. [ 0.057410] smp: Bringing up secondary CPUs ... [ 0.058537] x86: Booting SMP configuration: [ 0.059022] .... node #0, CPUs: #1 #2 #3 [ 0.063182] smp: Brought up 1 node, 4 CPUs [ 0.065015] smpboot: Max logical packages: 1 [ 0.066011] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.198916] node 0 deferred pages initialised in 129ms [ 0.200129] devtmpfs: initialized [ 0.203115] x86/mm: Memory block size: 128MB [ 0.207283] gcov: version magic: 0x41383552 [ 0.210371] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.215125] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.218369] pinctrl core: initialized pinctrl subsystem [ 0.220368] [ 0.221011] ************************************************************* [ 0.224018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.226021] ** ** [ 0.228019] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.231017] ** ** [ 0.233015] ** This means that this kernel is built to expose internal ** [ 0.236017] ** IOMMU data structures, which may compromise security on ** [ 0.238025] ** your system. ** [ 0.241018] ** ** [ 0.243019] ** If you see this message and you are not debugging the ** [ 0.246015] ** kernel, report this immediately to your vendor! ** [ 0.248018] ** ** [ 0.251016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.253026] ************************************************************* [ 0.257047] NET: Registered protocol family 16 [ 0.259861] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.264078] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.267081] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.273058] cpuidle: using governor menu [ 0.275000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.277011] PCI: Using configuration type 1 for base access [ 0.285131] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.293623] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.294042] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.296295] cryptd: max_cpu_qlen set to 1000 [ 0.298296] ACPI: Added _OSI(Module Device) [ 0.299021] ACPI: Added _OSI(Processor Device) [ 0.303021] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.306025] ACPI: Added _OSI(Processor Aggregator Device) [ 0.314906] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.319678] ACPI: Interpreter enabled [ 0.320133] ACPI: PM: (supports S0 S3 S4 S5) [ 0.322015] ACPI: Using IOAPIC for interrupt routing [ 0.323230] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.327000] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.345306] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.348044] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.352033] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.359074] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.364237] acpiphp: Slot [2] registered [ 0.365307] acpiphp: Slot [5] registered [ 0.367893] acpiphp: Slot [6] registered [ 0.369147] acpiphp: Slot [3] registered [ 0.371166] acpiphp: Slot [4] registered [ 0.372109] acpiphp: Slot [7] registered [ 0.374116] acpiphp: Slot [8] registered [ 0.375114] acpiphp: Slot [9] registered [ 0.377106] acpiphp: Slot [10] registered [ 0.379135] acpiphp: Slot [11] registered [ 0.380100] acpiphp: Slot [12] registered [ 0.382110] acpiphp: Slot [13] registered [ 0.384117] acpiphp: Slot [14] registered [ 0.385109] acpiphp: Slot [15] registered [ 0.387142] acpiphp: Slot [16] registered [ 0.395187] acpiphp: Slot [17] registered [ 0.397183] acpiphp: Slot [18] registered [ 0.401000] acpiphp: Slot [19] registered [ 0.403122] acpiphp: Slot [20] registered [ 0.404122] acpiphp: Slot [21] registered [ 0.406139] acpiphp: Slot [22] registered [ 0.409160] acpiphp: Slot [23] registered [ 0.411207] acpiphp: Slot [24] registered [ 0.413139] acpiphp: Slot [25] registered [ 0.415397] acpiphp: Slot [26] registered [ 0.417145] acpiphp: Slot [27] registered [ 0.418080] acpiphp: Slot [28] registered [ 0.419103] acpiphp: Slot [29] registered [ 0.421168] acpiphp: Slot [30] registered [ 0.422091] acpiphp: Slot [31] registered [ 0.423078] PCI host bridge to bus 0000:00 [ 0.424020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.426030] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.429033] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.431031] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.433029] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.436042] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.438197] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.440941] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.444262] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.451534] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.454909] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.459032] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.461027] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.464028] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.466615] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.469791] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.471052] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.473710] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.476739] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.484023] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.489019] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.501463] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.511019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.521022] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.536024] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.545000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.556014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.598013] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.610915] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.617729] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.620597] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.623447] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.628780] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.631366] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.640060] iommu: Default domain type: Passthrough [ 0.642986] SCSI subsystem initialized [ 0.644247] ACPI: bus type USB registered [ 0.646213] usbcore: registered new interface driver usbfs [ 0.649104] usbcore: registered new interface driver hub [ 0.651105] usbcore: registered new device driver usb [ 0.653191] pps_core: LinuxPPS API ver. 1 registered [ 0.655018] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.658129] PTP clock support registered [ 0.661058] EDAC MC: Ver: 3.0.0 [ 0.663184] PCI: Using ACPI for IRQ routing [ 0.664754] NetLabel: Initializing [ 0.665010] NetLabel: domain hash size = 128 [ 0.666009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.667094] NetLabel: unlabeled traffic allowed by default [ 0.669162] vgaarb: loaded [ 0.671367] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.673015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.680345] clocksource: Switched to clocksource kvm-clock [ 0.794853] VFS: Disk quotas dquot_6.6.0 [ 0.796460] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.798821] *** VALIDATE ramfs *** [ 0.799929] *** VALIDATE hugetlbfs *** [ 0.801525] pnp: PnP ACPI init [ 0.804443] pnp: PnP ACPI: found 6 devices [ 0.819671] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.822357] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.824089] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.826355] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.828587] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.830816] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.833708] NET: Registered protocol family 2 [ 0.836241] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.841238] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.846956] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.852613] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.856427] TCP: Hash tables configured (established 65536 bind 65536) [ 0.859526] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.862940] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.865879] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.869257] NET: Registered protocol family 1 [ 0.871831] RPC: Registered named UNIX socket transport module. [ 0.873853] RPC: Registered udp transport module. [ 0.879163] RPC: Registered tcp transport module. [ 0.880953] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.883386] NET: Registered protocol family 44 [ 0.885099] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.887224] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.889231] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.891581] PCI: CLS 0 bytes, default 64 [ 0.893221] Unpacking initramfs... [ 2.526661] debug: unmapping init [mem 0xffff9679fcc64000-0xffff9679fffcffff] [ 2.531127] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.533586] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.536500] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.037388] Initialise system trusted keyrings [ 3.039094] Key type blacklist registered [ 3.040865] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.049739] zbud: loaded [ 3.052572] *** VALIDATE nfs *** [ 3.053745] *** VALIDATE nfs4 *** [ 3.055431] pstore: using deflate compression [ 3.059271] Platform Keyring initialized [ 3.168621] NET: Registered protocol family 38 [ 3.170690] Key type asymmetric registered [ 3.172587] Asymmetric key parser 'x509' registered [ 3.174626] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.178161] io scheduler mq-deadline registered [ 3.179838] io scheduler kyber registered [ 3.181756] io scheduler bfq registered [ 3.183403] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.186768] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.190979] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.194188] ACPI: Power Button [PWRF] [ 3.199217] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.214699] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.313230] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.346903] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.382311] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.386685] Non-volatile memory driver v1.3 [ 3.388151] Linux agpgart interface v0.103 [ 3.419313] virtio_blk virtio1: [vda] 145160 512-byte logical blocks (74.3 MB/70.9 MiB) [ 3.421770] vda: detected capacity change from 0 to 74321920 [ 3.446548] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.449407] vdb: detected capacity change from 0 to 1073741824 [ 3.458834] libphy: Fixed MDIO Bus: probed [ 3.465178] usbcore: registered new interface driver usbserial_generic [ 3.467510] usbserial: USB Serial support registered for generic [ 3.469545] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.474755] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.476476] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.478940] mousedev: PS/2 mouse device common for all mice [ 3.487441] rtc_cmos 00:05: RTC can wake from S4 [ 3.489898] rtc_cmos 00:05: registered as rtc0 [ 3.491332] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.494442] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.495506] intel_pstate: CPU model not supported [ 3.500143] hid: raw HID events driver (C) Jiri Kosina [ 3.503751] usbcore: registered new interface driver usbhid [ 3.505930] usbhid: USB HID core driver [ 3.507451] drop_monitor: Initializing network drop monitor service [ 3.509985] Initializing XFRM netlink socket [ 3.512601] NET: Registered protocol family 10 [ 3.512797] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.515779] Segment Routing with IPv6 [ 3.518058] NET: Registered protocol family 17 [ 3.519918] mpls_gso: MPLS GSO support [ 3.520182] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.528213] RAS: Correctable Errors collector initialized. [ 3.529889] AVX version of gcm_enc/dec engaged. [ 3.531537] AES CTR mode by8 optimization enabled [ 3.621823] sched_clock: Marking stable (3621741453, 0)->(4563778842, -942037389) [ 3.625666] registered taskstats version 1 [ 3.627753] Loading compiled-in X.509 certificates [ 3.630694] zswap: loaded using pool lzo/zbud [ 3.668636] Key type big_key registered [ 3.689720] Key type encrypted registered [ 3.691210] ima: No TPM chip found, activating TPM-bypass! [ 3.693286] ima: Allocated hash algorithm: sha1 [ 3.695019] ima: No architecture policies found [ 3.696738] evm: Initialising EVM extended attributes: [ 3.698781] evm: security.selinux [ 3.700367] evm: security.ima [ 3.701910] evm: security.capability [ 3.703771] evm: HMAC attrs: 0x1 [ 3.706626] rtc_cmos 00:05: setting system clock to 2026-07-03 09:31:40 UTC (1783071100) [ 3.712982] debug: unmapping init [mem 0xffffffffb6403000-0xffffffffb65fffff] [ 3.715468] debug: unmapping init [mem 0xffffffffb5182000-0xffffffffb5458fff] [ 3.723322] Write protecting the kernel read-only data: 28672k [ 3.727331] debug: unmapping init [mem 0xffffffffb3803000-0xffffffffb39fffff] [ 3.730721] debug: unmapping init [mem 0xffffffffb4114000-0xffffffffb41fffff] [ 3.767260] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.777898] systemd[1]: Detected virtualization kvm. [ 3.780818] systemd[1]: Detected architecture x86-64. [ 3.784227] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.819293] systemd[1]: No hostname configured. [ 3.821388] systemd[1]: Set hostname to . [ 3.823990] random: systemd: uninitialized urandom read (16 bytes read) [ 3.827257] systemd[1]: Initializing machine ID from random generator. [ 3.880688] random: ln: uninitialized urandom read (6 bytes read) [ 4.005159] random: systemd: uninitialized urandom read (16 bytes read) [ 4.009384] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 4.047425] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 4.067686] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Slices. [ OK ] Reached target Paths. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... [ OK ] Reached target Sockets. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.171877] device-mapper: uevent: version 1.0.3 [ 5.174549] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 6.958464] virtio_net virtio0 ens2: renamed from eth0 [ 7.008180] random: fast init done [ 7.167448] scsi host0: ata_piix [ 7.190158] scsi host1: ata_piix [ 7.290337] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 7.298905] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 12.217610] random: crng init done [ 12.219182] random: 7 urandom warning(s) missed due to ratelimiting [ 13.631193] dracut-initqueue[588]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 15.938919] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 20.140640] printk: systemd: 26 output lines suppressed due to ratelimiting [ 21.356263] SELinux: Disabled at runtime. [ 21.499735] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 21.524243] systemd[1]: Detected virtualization kvm. [ 21.531649] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 24.146160] systemd[1]: initrd-switch-root.service: Succeeded. [ 24.163893] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 24.189161] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 24.208041] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 24.237646] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 24.273719] systemd[1]: Starting Journal Service... Starting Journal Service... [ 24.345853] systemd[1]: Created slice User and Session Slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-sshd\x2dkeygen.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Slices. Mounting POSIX Message Queue File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-serial\x2dgetty.slice. Activating swap /dev/disk/by-label/SWAP... Mounting Kernel Debug File System... Starting Remount Root and Kernel File Systems... [ 24.759466] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-getty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Process Core Dump Socket. Mounting Huge Pages File System... [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Initrd Root File System. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 26.845874] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 29.103015] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 29.213315] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 30.615754] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 30.863657] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (9s / no limit) [** ] A start job is running for Configur…-only root support (9s / no limit) [*** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (11s / no limit) [ ***] A start job is running for Configur…only root support (11s / no limit) [ **] A start job is running for Configur…only root support (12s / no limit) [ *] A start job is running for Configur…only root support (12s / no limit) [ **] A start job is running for Configur…only root support (13s / no limit) [ ***] A start job is running for Configur…only root support (14s / no limit) [ *** ] A start job is running for Configur…only root support (14s / no limit)[ 38.999990] Key type dns_resolver registered [ *** ] A start job is running for Configur…only root support (15s / no limit) [*** ] A start job is running for Configur…only root support (15s / no limit) [** ] A start job is running for Configur…only root support (16s / no limit)[ 40.387472] NFS: Registering the id_resolver key type [ 40.391356] Key type id_resolver registered [ 40.393153] Key type id_legacy registered [* ] A start job is running for Configur…only root support (16s / no limit) [** ] A start job is running for Configur…only root support (17s / no limit) [*** ] A start job is running for Configur…only root support (17s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. Starting Network Manager... Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Login Service. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg318-client login: [ 110.150873] libcfs: loading out-of-tree module taints kernel. [ 110.235226] Key type ._llcrypt registered [ 110.254689] Key type .llcrypt registered [ 111.094505] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 111.107349] alg: No test for adler32 (adler32-zlib) [ 112.645894] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 113.798804] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 115.759150] Key type lgssc registered [ 117.800327] Lustre: Echo OBD driver; http://www.lustre.org/ [ 146.229124] hrtimer: interrupt took 6620766 ns [ 226.717280] Lustre: Mounted lustre-client [ 231.313529] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 243.374713] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing check_logdir /tmp/testlogs/ [ 247.164074] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing yml_node [ 250.820876] Lustre: DEBUG MARKER: Client: 2.17.54.83 [ 252.392666] Lustre: lustre-OST0000-osc-ffff967a504ac000: disconnect after 24s idle [ 253.297724] Lustre: DEBUG MARKER: MDS: 2.17.54.83 [ 255.424357] Lustre: DEBUG MARKER: OSS: 2.17.54.83 [ 257.040424] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Fri Jul 3 05:35:52 EDT 2026 [ 272.816420] Lustre: Unmounted lustre-client [ 292.803936] Key type lgssc unregistered [ 293.021683] LNet: 8384:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 293.028499] LNetError: 8384:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 293.043503] LNet: Removed LNI 192.168.203.18@tcp [ 293.746215] Key type .llcrypt unregistered [ 293.749630] Key type ._llcrypt unregistered [ 316.017600] Key type ._llcrypt registered [ 316.019628] Key type .llcrypt registered [ 316.078066] LNet: 8963:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 316.078066] mask usage: [+|-] ... [ 316.265601] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 316.280489] alg: No test for adler32 (adler32-zlib) [ 317.194913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 317.257658] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 317.312189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 321.112891] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 321.189285] LNet: 9259:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 321.200307] LNetError: 9259:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 321.230075] LNet: Removed LNI 192.168.203.18@tcp [ 321.782153] Key type .llcrypt unregistered [ 321.785783] Key type ._llcrypt unregistered [ 327.563626] Key type ._llcrypt registered [ 327.565803] Key type .llcrypt registered [ 328.652851] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 05:37:04 (1783071424) [ 328.948220] LNet: 9640:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 328.948220] mask usage: [+|-] ... [ 329.112501] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 329.125027] alg: No test for adler32 (adler32-zlib) [ 330.067526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 330.147357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 330.179440] LNet: 9678:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 330.222800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 330.290959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 330.316068] LNet: 9685:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 336.991461] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 05:37:12 (1783071432) [ 337.082214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 337.113280] LNet: 10256:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 337.132800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 337.211779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 344.570119] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 05:37:19 (1783071439) [ 344.719809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 344.753641] LNet: 10846:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 344.774792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 344.890311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 351.953089] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 05:37:27 (1783071447) [ 352.105441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 352.162369] LNet: 11433:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 352.213941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 352.321199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 360.403034] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 05:37:35 (1783071455) [ 360.750829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 360.781422] LNet: 12021:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 360.822427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 360.907339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 361.760589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 370.692427] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 05:37:45 (1783071465) [ 370.740438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 370.754771] LNet: 12605:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 370.784351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 370.901802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 371.602896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 372.353380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 381.083821] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 05:37:55 (1783071475) [ 381.288084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 381.359243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 381.485628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 390.027974] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 05:38:05 (1783071485) [ 390.194050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 390.239120] LNet: 13786:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 390.251107] LNet: 13786:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 390.295412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 390.400860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 390.432944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 391.259312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 391.334942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 391.904424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 391.989856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 392.776471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 392.845310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 393.514971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 393.603716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 394.298616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 394.338362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 394.986659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 395.040796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 395.636595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 395.692912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 396.447811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 396.480420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 396.931644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 396.984109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 397.530804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 397.585456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 398.127250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 398.183838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 398.784384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 398.866206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 406.805544] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 05:38:22 (1783071502) [ 406.902168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 406.973642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 407.054233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 407.085239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 413.373884] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 05:38:29 (1783071509) [ 413.516817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 413.574623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 413.721676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 413.776798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 420.928461] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 05:38:36 (1783071516) [ 421.124465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 421.251608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 421.370733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 421.432246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 428.499967] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 05:38:43 (1783071523) [ 428.670967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 428.696710] LNet: 16235:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 428.711779] LNet: 16235:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 428.733958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 428.804214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 428.833541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 435.653857] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 05:38:51 (1783071531) [ 435.830980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 435.898959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 435.976976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 436.036476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 442.977874] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 05:38:58 (1783071538) [ 443.069620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 443.124217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 443.214192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 443.241480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 450.180100] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 05:39:05 (1783071545) [ 450.366868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 450.423680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 450.507399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 450.994747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 456.823125] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 05:39:12 (1783071552) [ 457.006696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 457.058786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 457.396971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 457.774440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 465.391508] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 05:39:20 (1783071560) [ 465.489866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 465.559424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 465.765591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 466.081922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 474.136940] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 05:39:29 (1783071569) [ 474.519927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 474.585686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 475.057439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 475.330741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 482.339665] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 05:39:37 (1783071577) [ 482.421568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 482.474901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 482.547957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 482.618522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 488.804622] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 05:39:44 (1783071584) [ 488.911829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 488.953955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 489.048697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 489.105856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 495.403782] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 05:39:50 (1783071590) [ 495.503465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 495.530139] LNet: 21672:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 495.535627] LNet: 21672:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 495.568341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 495.652720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 502.392543] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 05:39:57 (1783071597) [ 502.496851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 502.578831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 502.702226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 509.707860] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 05:40:05 (1783071605) [ 509.820065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 509.861992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 509.968473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 510.298623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 517.164730] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 05:40:12 (1783071612) [ 517.415201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 517.454264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 517.530871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 517.583625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 524.199857] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 05:40:19 (1783071619) [ 524.304081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 524.405543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 524.526260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 524.626730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 532.045614] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 05:40:27 (1783071627) [ 532.315919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 532.353850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 532.421568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 532.473726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 539.846568] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 05:40:34 (1783071634) [ 539.950581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 539.994919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 540.075792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 540.126382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 546.295301] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 05:40:41 (1783071641) [ 546.416566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 546.466126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 546.533191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 546.604050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 546.629905] LNetError: 25796:0:(peer.c:2234:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 546.682718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 546.715641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 552.754757] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 05:40:48 (1783071648) [ 552.909829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 552.965063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 553.099806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.8F6jHML2al [ 553.216360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.8F6jHML2al [ 553.299594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.8F6jHML2al [ 558.866973] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 05:40:54 (1783071654) [ 559.065340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 559.097894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 559.140341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 559.171694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 559.209028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 564.370868] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 05:40:59 (1783071659) [ 564.457888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 564.497844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 564.583346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 566.761533] LNet: Message forwarding will be enabled in 2s [ 569.823577] LNet: Message forwarding enabled [ 570.255787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 570.279792] LNet: Message forwarding will stop in 2s to 6s [ 574.762082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 575.354169] LNet: Message forwarding disabled [ 576.803935] LNet: Message forwarding will be enabled in 2s [ 580.511098] LNet: Message forwarding enabled [ 582.125762] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 05:41:17 (1783071677) [ 582.244621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 583.063485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 583.114759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 588.714574] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 05:41:24 (1783071684) [ 588.832701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 588.861878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 588.910454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 588.951885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 588.984924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 589.027356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 589.059957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 589.102113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 589.147247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 589.181080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 589.221725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 589.261758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 589.300820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 589.331733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 589.364668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 589.407694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 589.431856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 589.459745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 589.498511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 589.536272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 589.595283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 589.625427] LNetError: 28865:0:(router.c:624:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 594.762755] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 05:41:30 (1783071690) [ 594.844088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 594.889272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 594.965600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 600.692319] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 05:41:36 (1783071696) [ 602.320815] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg318-server [ 604.139873] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 05:41:39 (1783071699) [ 604.188289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 604.217824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 604.286744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 604.324792] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 605.716392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.203.[19-23]@tcp [ 605.735443] LNet: 30581:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 605.798532] LNetError: 30367:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.21@tcp is being used as a gateway but routing feature is not turned on [ 605.811568] LNetError: 30367:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.21@tcp has gone from up to down [ 606.761102] LNetError: 30367:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.20@tcp is being used as a gateway but routing feature is not turned on [ 606.774222] LNetError: 30367:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 7 previous similar messages [ 607.785666] LNetError: 30367:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.20@tcp is being used as a gateway but routing feature is not turned on [ 607.798459] LNetError: 30367:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 3 previous similar messages [ 608.865688] LNetError: 30583:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.22@tcp has gone from up to down [ 608.877260] LNetError: 30583:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) Skipped 3 previous similar messages [ 609.827960] LNetError: 30367:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.23@tcp is being used as a gateway but routing feature is not turned on [ 609.838899] LNetError: 30367:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 7 previous similar messages [ 610.916886] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 05:41:46 (1783071706) [ 612.506896] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg318-server [ 613.931087] LNetError: 30367:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.20@tcp is being used as a gateway but routing feature is not turned on [ 613.952353] LNetError: 30367:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 15 previous similar messages [ 614.069852] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 05:41:49 (1783071709) [ 614.179547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 614.943935] LNetError: 30365:0:(router.c:1266:lnet_check_routers()) Failed to discover router 192.168.203.19@tcp [ 615.268699] LNetError: 31514:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 615.291787] LNet: Removed LNI 192.168.203.18@tcp [ 615.326470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 615.397067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 615.424483] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 616.045788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.203.[19-24/2]@tcp [ 616.082626] LNet: 31633:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 616.094269] LNet: 31633:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 616.113543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.203.[19-24/2]@tcp [ 616.133322] LNetError: 31520:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp103 through 192.168.203.23@tcp has gone from up to down [ 621.501453] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 05:41:57 (1783071717) [ 621.757164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 621.797221] LNetError: 32225:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 621.820529] LNet: Removed LNI 192.168.203.18@tcp [ 621.840938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 621.958405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 622.041353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 622.081675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 622.145375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 622.198534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 622.234799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 622.273417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 622.336564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 622.359759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 622.403543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 622.477723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 622.502817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 622.542387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 622.619036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 622.676817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 622.747340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 622.809630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 622.836238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 622.878437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 622.950844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 622.986716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 623.056055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 623.129408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 623.165042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 623.217402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 623.290985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 623.326681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 623.388652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 623.449712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 623.481288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 623.531980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 623.578963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 623.609689] LNet: 32372:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 623.620058] LNet: 32372:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 26 previous similar messages [ 623.640989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 623.692756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 628.547285] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 05:42:04 (1783071724) [ 630.262401] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg318-server [ 631.636647] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 05:42:07 (1783071727) [ 633.256966] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg318-server [ 634.635817] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 05:42:10 (1783071730) [ 635.421301] Key type .llcrypt unregistered [ 635.430337] Key type ._llcrypt unregistered [ 640.425777] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 640.630292] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 640.639752] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 640.647235] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 643.126696] Key type ._llcrypt registered [ 643.129123] Key type .llcrypt registered [ 643.167627] LNet: 33908:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 643.167627] mask usage: [+|-] ... [ 643.332789] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 643.355555] alg: No test for adler32 (adler32-zlib) [ 644.241728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 644.287868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 644.307432] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 644.959457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 644.981325] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 646.789585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 646.822432] LNet: Removed LNI 10.1.2.3@tcp [ 646.928881] LNet: 34352:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 646.946849] LNetError: 34352:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 646.968061] LNet: Removed LNI 192.168.203.18@tcp [ 647.533316] Key type .llcrypt unregistered [ 647.537818] Key type ._llcrypt unregistered [ 659.831849] Key type ._llcrypt registered [ 659.834315] Key type .llcrypt registered [ 660.923389] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 05:42:36 (1783071756) [ 661.555293] Key type .llcrypt unregistered [ 661.557869] Key type ._llcrypt unregistered [ 666.927140] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 667.104836] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 667.115512] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 667.118516] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 669.572454] Key type ._llcrypt registered [ 669.574451] Key type .llcrypt registered [ 669.596965] LNet: 35384:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 669.596965] mask usage: [+|-] ... [ 669.782272] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 669.802166] alg: No test for adler32 (adler32-zlib) [ 670.762944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 670.831878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 670.863446] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 671.561386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 671.586211] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 674.059578] LNet: 35842:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 674.072296] LNetError: 35842:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 674.078199] LNet: Removed LNI 10.1.2.3@tcp [ 674.086463] LNet: Removed LNI 192.168.203.18@tcp [ 674.728486] Key type .llcrypt unregistered [ 674.730967] Key type ._llcrypt unregistered [ 684.568632] Key type ._llcrypt registered [ 684.571517] Key type .llcrypt registered [ 685.691980] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 05:43:01 (1783071781) [ 686.432830] Key type .llcrypt unregistered [ 686.444544] Key type ._llcrypt unregistered [ 697.640183] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 698.055336] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 698.062736] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 698.078722] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 701.016572] Key type ._llcrypt registered [ 701.020136] Key type .llcrypt registered [ 701.046324] LNet: 36877:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 701.046324] mask usage: [+|-] ... [ 701.185826] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 701.212967] alg: No test for adler32 (adler32-zlib) [ 702.092912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 702.141957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 702.161591] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 704.341942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 704.357916] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 708.087965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 708.119388] LNet: Removed LNI 10.1.2.3@tcp [ 708.166720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 708.267958] LNet: Removed LNI 10.1.2.31@tcp [ 708.309915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 708.380576] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 709.999405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 710.041259] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 711.983441] LNet: 37761:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 711.989996] LNetError: 37761:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 711.992747] LNet: Removed LNI 10.1.2.31@tcp [ 712.015131] LNet: Removed LNI 10.1.2.3@tcp [ 712.887466] Key type .llcrypt unregistered [ 712.892304] Key type ._llcrypt unregistered [ 728.444890] Key type ._llcrypt registered [ 728.451770] Key type .llcrypt registered [ 730.149235] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 05:43:45 (1783071825) [ 730.360647] LNet: 38555:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 730.360647] mask usage: [+|-] ... [ 730.501463] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 730.519714] alg: No test for adler32 (adler32-zlib) [ 731.405480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 731.467824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 731.494054] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 732.165765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 732.179770] LNet: 38716:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 732.187622] LNetError: 38716:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 732.199315] LNet: Removed LNI 192.168.203.18@tcp [ 732.215050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 732.250552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.203.18@tcp [ 732.268613] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 732.766928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.203.18@tcp [ 732.799995] LNet: Removed LNI 192.168.203.18@tcp [ 732.853655] LNet: 38851:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 733.461373] Key type .llcrypt unregistered [ 733.464379] Key type ._llcrypt unregistered [ 745.140178] Key type ._llcrypt registered [ 745.142932] Key type .llcrypt registered [ 746.863097] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 05:44:01 (1783071841) [ 748.780492] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg318-server [ 750.380688] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 05:44:05 (1783071845) [ 750.927386] Key type .llcrypt unregistered [ 750.929086] Key type ._llcrypt unregistered [ 756.884316] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 757.142178] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 757.156480] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 757.167379] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 758.687173] Key type ._llcrypt registered [ 758.693058] Key type .llcrypt registered [ 758.744314] LNet: 40221:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 758.744314] mask usage: [+|-] ... [ 759.043469] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 759.080120] alg: No test for adler32 (adler32-zlib) [ 760.030882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 760.134899] LNet: Added LNI 192.168.203.18@tcp1 [8/256/0/180] [ 761.088634] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 763.208881] LNet: 40680:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 763.230966] LNetError: 40680:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 763.242288] LNet: Removed LNI 192.168.203.18@tcp1 [ 763.270252] LNet: Removed LNI 10.1.2.3@tcp2 [ 763.980292] Key type .llcrypt unregistered [ 763.982124] Key type ._llcrypt unregistered [ 774.517494] Key type ._llcrypt registered [ 774.521776] Key type .llcrypt registered [ 775.749673] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 05:44:31 (1783071871) [ 775.986759] LNet: 41484:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 775.986759] mask usage: [+|-] ... [ 776.117311] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 776.127465] alg: No test for adler32 (adler32-zlib) [ 777.052479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 777.092852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 777.122255] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 777.746907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 777.772896] LNet: 41649:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 777.778325] LNetError: 41649:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 777.801564] LNet: Removed LNI 192.168.203.18@tcp [ 777.822497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 777.890692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 777.921038] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 784.088995] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 05:44:39 (1783071879) [ 784.305504] LNet: 42240:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 784.310816] LNetError: 42240:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 784.330945] LNet: Removed LNI 192.168.203.18@tcp [ 785.094549] Key type .llcrypt unregistered [ 785.100121] Key type ._llcrypt unregistered [ 790.789483] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 790.914200] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 790.922582] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 790.929900] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 793.519930] Key type ._llcrypt registered [ 793.527152] Key type .llcrypt registered [ 793.597768] LNet: 42518:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 793.597768] mask usage: [+|-] ... [ 793.810704] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 793.836422] alg: No test for adler32 (adler32-zlib) [ 794.793351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 794.864276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 794.900135] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 795.633159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 795.653989] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 797.476352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 797.488650] LNet: 42984:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 797.507392] LNetError: 42984:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 797.513881] LNet: Removed LNI 10.1.2.3@tcp [ 797.546687] LNet: Removed LNI 192.168.203.18@tcp [ 797.575770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 797.657954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 797.692550] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 797.698754] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 798.194516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 798.210880] LNet: 43011:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 798.218467] LNetError: 43011:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 798.228216] LNetError: 43011:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 798.244522] LNet: Removed LNI 10.1.2.3@tcp [ 798.254211] LNet: Removed LNI 192.168.203.18@tcp [ 798.268488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 798.317436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 798.335581] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 798.339554] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 798.809752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 798.842424] LNet: Added LNI 192.168.203.18@tcp2 [8/256/0/180] [ 799.685495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 799.713157] LNet: 43173:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 799.719800] LNetError: 43173:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 799.735123] LNetError: 43173:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 799.750230] LNet: Removed LNI 10.1.2.3@tcp [ 799.753408] LNet: Removed LNI 192.168.203.18@tcp [ 799.772815] LNet: Removed LNI 192.168.203.18@tcp2 [ 799.808560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 799.878082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 799.911626] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 799.916508] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 799.925613] LNet: Added LNI 192.168.203.18@tcp2 [8/256/0/180] [ 806.506264] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 05:45:01 (1783071901) [ 806.590269] LNet: 43761:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 806.597206] LNetError: 43761:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 806.609394] LNetError: 43761:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 806.615651] LNet: Removed LNI 10.1.2.3@tcp [ 806.618375] LNet: Removed LNI 192.168.203.18@tcp [ 806.654622] LNet: Removed LNI 192.168.203.18@tcp2 [ 807.284532] Key type .llcrypt unregistered [ 807.289037] Key type ._llcrypt unregistered [ 813.258538] Key type ._llcrypt registered [ 813.261399] Key type .llcrypt registered [ 813.300600] LNet: 43997:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 813.300600] mask usage: [+|-] ... [ 813.447374] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 813.461739] alg: No test for adler32 (adler32-zlib) [ 814.407795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 814.451617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 814.475728] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 815.267714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 815.296558] LNet: 44166:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 815.303397] LNetError: 44166:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 815.323984] LNet: Removed LNI 192.168.203.18@tcp [ 815.355129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 815.406912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 815.429694] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 815.807326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 815.831370] LNet: 44196:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 815.836139] LNetError: 44196:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 815.856392] LNet: Removed LNI 192.168.203.18@tcp [ 815.880857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 815.972098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 816.007649] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 816.410906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 816.438886] LNet: Removed LNI 192.168.203.18@tcp [ 816.454543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 816.522539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 816.548477] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 816.996111] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 817.106887] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 817.111216] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 817.116509] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 819.611899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 819.636866] LNet: 44304:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 819.640495] LNet: 44304:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 819.651386] LNetError: 44304:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 819.656700] LNetError: 44304:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 819.669891] LNet: Removed LNI 192.168.203.18@tcp [ 819.699800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 819.795844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 819.825315] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 826.383020] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 05:45:21 (1783071921) [ 826.604691] LNet: 44901:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 826.612935] LNetError: 44901:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 826.638508] LNet: Removed LNI 192.168.203.18@tcp [ 827.249811] Key type .llcrypt unregistered [ 827.253887] Key type ._llcrypt unregistered [ 833.331885] Key type ._llcrypt registered [ 833.338290] Key type .llcrypt registered [ 833.390308] LNet: 45128:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 833.390308] mask usage: [+|-] ... [ 833.557108] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 833.573586] alg: No test for adler32 (adler32-zlib) [ 834.531191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 834.595089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 834.624767] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 835.594800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 835.631251] LNet: 45301:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 835.635121] LNetError: 45301:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 835.652027] LNet: Removed LNI 192.168.203.18@tcp [ 835.670062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 835.758601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 835.794577] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 842.557217] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 05:45:38 (1783071938) [ 842.745385] LNet: 45892:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 842.759147] LNetError: 45892:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 842.798730] LNet: Removed LNI 192.168.203.18@tcp [ 843.522342] Key type .llcrypt unregistered [ 843.525139] Key type ._llcrypt unregistered [ 849.185540] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 849.344695] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 849.353774] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 849.361367] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 852.220847] Key type ._llcrypt registered [ 852.225902] Key type .llcrypt registered [ 852.300861] LNet: 46173:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 852.300861] mask usage: [+|-] ... [ 852.535220] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 852.567496] alg: No test for adler32 (adler32-zlib) [ 853.465100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 853.524988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 853.560157] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 854.342338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 854.358393] LNet: 46346:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 854.366650] LNetError: 46346:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 854.377505] LNet: Removed LNI 192.168.203.18@tcp [ 854.403111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 854.480779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 854.507794] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 860.937047] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 05:45:56 (1783071956) [ 861.035977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 861.067488] LNet: 46937:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 861.074598] LNetError: 46937:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 861.099062] LNet: Removed LNI 192.168.203.18@tcp [ 861.164194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 861.276830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 867.073808] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 05:46:02 (1783071962) [ 867.213545] LNet: 47522:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 867.950256] Key type .llcrypt unregistered [ 867.955238] Key type ._llcrypt unregistered [ 873.786186] Key type ._llcrypt registered [ 873.789246] Key type .llcrypt registered [ 873.848792] LNet: 47752:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 873.848792] mask usage: [+|-] ... [ 874.052113] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 874.074473] alg: No test for adler32 (adler32-zlib) [ 875.043293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 875.104802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 875.153967] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 876.000355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 876.017329] LNet: 47924:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 876.023253] LNetError: 47924:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 876.043810] LNet: Removed LNI 192.168.203.18@tcp [ 876.059317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 876.135632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 876.154503] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 882.603273] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 05:46:18 (1783071978) [ 882.707851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 882.746174] LNet: 48511:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 882.752812] LNetError: 48511:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 882.773456] LNet: Removed LNI 192.168.203.18@tcp [ 882.793682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 882.881702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 888.120279] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 05:46:23 (1783071983) [ 889.534945] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 890.896937] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 05:46:26 (1783071986) [ 892.199754] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 893.488636] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 05:46:29 (1783071989) [ 894.861823] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 896.698356] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 05:46:32 (1783071992) [ 897.924611] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 899.712567] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 05:46:35 (1783071995) [ 900.998265] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 902.470445] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 05:46:38 (1783071998) [ 902.707809] LNet: 50570:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 903.257380] Key type .llcrypt unregistered [ 903.261364] Key type ._llcrypt unregistered [ 909.173572] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 909.366526] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 909.374383] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 909.377183] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 912.005959] Key type ._llcrypt registered [ 912.011456] Key type .llcrypt registered [ 912.054444] LNet: 50852:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 912.054444] mask usage: [+|-] ... [ 912.247607] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 912.261548] alg: No test for adler32 (adler32-zlib) [ 913.125643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 913.205500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 913.228567] LNet: Added LNI 192.168.203.18@tcp [8/512/0/180] [ 913.233480] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 913.236846] LNet: 50891:0:(api-ni.c:2914:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 913.320465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 913.342974] LNet: 50910:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 913.350422] LNetError: 50910:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 913.355281] LNet: Removed LNI 10.1.2.3@tcp [ 913.358417] LNetError: 50910:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 913.370813] LNet: Removed LNI 192.168.203.18@tcp [ 913.387833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 913.453091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 913.480314] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 913.562653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 913.577543] LNet: 50934:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 913.585783] LNetError: 50934:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 913.606649] LNet: Removed LNI 192.168.203.18@tcp [ 913.625886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 913.678362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 913.693074] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 913.699843] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 913.706041] LNet: 50942:0:(api-ni.c:2914:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 913.777747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 913.796442] LNet: 50960:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 913.801290] LNetError: 50960:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 913.807733] LNet: Removed LNI 10.1.2.3@tcp [ 913.810244] LNetError: 50960:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 913.821724] LNet: Removed LNI 192.168.203.18@tcp [ 913.837568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 913.904541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 913.950285] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 913.956690] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 913.958921] LNet: 50968:0:(api-ni.c:2914:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 914.122653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 914.138347] LNet: 50994:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 914.144930] LNetError: 50994:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 914.149798] LNet: Removed LNI 10.1.2.3@tcp [ 914.151824] LNetError: 50994:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 914.163785] LNet: Removed LNI 192.168.203.18@tcp [ 914.190451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 914.231800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 914.249180] LNet: Added LNI 192.168.203.18@tcp [8/256/0/182] [ 914.252949] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 914.342696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 914.358254] LNet: 51022:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 914.363833] LNetError: 51022:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 914.370454] LNet: Removed LNI 10.1.2.3@tcp [ 914.372093] LNetError: 51022:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 914.388073] LNet: Removed LNI 192.168.203.18@tcp [ 914.413662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 914.487529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 914.526731] LNet: Added LNI 192.168.203.18@tcp [8/256/0/182] [ 914.540571] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 914.543943] LNet: 51030:0:(api-ni.c:2914:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 914.611883] LNet: Removed LNI 10.1.2.3@tcp [ 914.622552] LNet: Removed LNI 192.168.203.18@tcp [ 915.153478] Key type .llcrypt unregistered [ 915.158624] Key type ._llcrypt unregistered [ 925.242943] Key type ._llcrypt registered [ 925.248143] Key type .llcrypt registered [ 926.567640] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 05:47:01 (1783072021) [ 926.763522] LNet: 51844:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 926.763522] mask usage: [+|-] ... [ 926.885589] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 926.905302] alg: No test for adler32 (adler32-zlib) [ 927.832533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 927.903233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 927.935283] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 928.607565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.203.18 [ 928.621844] LNet: 52005:0:(config.c:1627:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 928.628706] LNet: Removed LNI 192.168.203.18@tcp [ 935.002414] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 05:47:10 (1783072030) [ 935.130483] LNet: 52568:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 935.156238] LNetError: 52568:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 935.185560] LNet: Removed LNI 192.168.203.18@tcp [ 935.922283] Key type .llcrypt unregistered [ 935.924810] Key type ._llcrypt unregistered [ 943.276293] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 943.607702] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 946.589922] Key type ._llcrypt registered [ 946.592282] Key type .llcrypt registered [ 946.629421] LNet: 52857:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 946.629421] mask usage: [+|-] ... [ 946.803919] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 946.819433] alg: No test for adler32 (adler32-zlib) [ 948.174345] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 955.917324] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 05:47:30 (1783072050) [ 956.055142] LNet: 53480:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 956.068798] LNetError: 53480:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 956.093804] LNet: Removed LNI 10.1.2.3@tcp [ 956.838539] Key type .llcrypt unregistered [ 956.840681] Key type ._llcrypt unregistered [ 964.411340] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 964.619307] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 964.622188] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 964.625733] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 967.454041] Key type ._llcrypt registered [ 967.464962] Key type .llcrypt registered [ 967.571139] LNet: 53761:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 967.571139] mask usage: [+|-] ... [ 967.940615] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 967.989449] alg: No test for adler32 (adler32-zlib) [ 969.145513] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 976.493618] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 05:47:51 (1783072071) [ 978.581505] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 980.384590] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 05:47:55 (1783072075) [ 980.594864] LNet: 54668:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 980.602446] LNetError: 54668:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 980.624346] LNet: Removed LNI 10.1.2.3@tcp [ 981.378642] Key type .llcrypt unregistered [ 981.385392] Key type ._llcrypt unregistered [ 987.918667] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 988.197791] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 988.208461] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 988.220872] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 989.867367] Key type ._llcrypt registered [ 989.870537] Key type .llcrypt registered [ 989.903604] LNet: 54939:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 989.903604] mask usage: [+|-] ... [ 990.084904] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 990.124279] alg: No test for adler32 (adler32-zlib) [ 991.155661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 991.211424] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 999.733411] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 05:48:15 (1783072095) [ 1000.271391] LNet: 55895:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1000.277742] LNetError: 55895:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1000.295473] LNet: Removed LNI 10.1.2.3@tcp [ 1001.007229] Key type .llcrypt unregistered [ 1001.009219] Key type ._llcrypt unregistered [ 1007.120972] Key type ._llcrypt registered [ 1007.125448] Key type .llcrypt registered [ 1007.211451] LNet: 56119:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1007.211451] mask usage: [+|-] ... [ 1007.428480] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1007.462354] alg: No test for adler32 (adler32-zlib) [ 1008.421636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1008.466909] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 1016.936511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.118@tcp [ 1017.521515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.18@tcp -d 192.168.203.118@tcp -m GET -e local_interrupt -r 1 [ 1017.550728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.18@tcp -d 192.168.203.18@tcp -m GET -e local_interrupt -r 1 [ 1017.584650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.118@tcp [ 1017.605406] LNet: There was an unexpected network error while writing to 192.168.203.118: rc = -22 [ 1017.695349] LNet: 1 local NIs in recovery (showing 1): 192.168.203.18@tcp [ 1017.817914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1017.840838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1017.868160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1017.894351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1017.920673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1017.946235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1017.985032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1018.062650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1018.083238] LNet: 56333:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1018.112870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1018.163805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1018.196924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1018.225070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1018.259713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1018.296705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1018.334953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1019.207701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.18@tcp -d 192.168.203.118@tcp -m GET -e local_dropped -r 1 [ 1019.311872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.18@tcp -d 192.168.203.18@tcp -m GET -e local_dropped -r 1 [ 1019.381938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.118@tcp [ 1019.437809] LNet: There was an unexpected network error while writing to 192.168.203.118: rc = -22 [ 1019.779653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1019.832909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1019.883336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1019.909898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1019.997707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1020.064413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1020.146655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1020.196841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1020.231866] LNet: 56442:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1020.261640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1020.323423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1020.362873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1020.398749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1020.450181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1020.513479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1020.553364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1021.402733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.18@tcp -d 192.168.203.118@tcp -m GET -e local_aborted -r 1 [ 1021.486919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.18@tcp -d 192.168.203.18@tcp -m GET -e local_aborted -r 1 [ 1021.546530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.118@tcp [ 1021.587714] LNet: There was an unexpected network error while writing to 192.168.203.118: rc = -22 [ 1021.882297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1021.939615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1021.986377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1022.015935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1022.058663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1022.102638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1022.142330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1022.170386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1022.214356] LNet: 56552:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1022.238600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1022.307708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1022.423345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1022.490363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1022.510060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1022.551775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1022.618529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1023.122519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.18@tcp -d 192.168.203.118@tcp -m GET -e local_no_route -r 1 [ 1023.160761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.18@tcp -d 192.168.203.18@tcp -m GET -e local_no_route -r 1 [ 1023.198767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.118@tcp [ 1023.469636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1023.515771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1023.601713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1023.645745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1023.677276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1023.731702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1023.799706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1023.831198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1023.892541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1023.971747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1024.031793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1024.078937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1024.116733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1024.144272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1024.184908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1024.561870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.18@tcp -d 192.168.203.118@tcp -m GET -e local_timeout -r 1 [ 1024.598892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.18@tcp -d 192.168.203.18@tcp -m GET -e local_timeout -r 1 [ 1024.632557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.118@tcp [ 1024.643372] LNet: There was an unexpected network error while writing to 192.168.203.118: rc = -22 [ 1024.647515] LNet: Skipped 1 previous similar message [ 1024.876496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1024.907085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1024.936480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1024.969155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1024.995562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1025.026395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1025.059992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1025.098678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1025.130846] LNet: 56770:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1025.143138] LNet: 56770:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1025.181782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1025.227712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1025.335712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1025.417901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1025.506700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1025.553572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1025.585047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1026.277252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.18@tcp -d 192.168.203.118@tcp -m GET -e local_error -r 1 [ 1026.360297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.18@tcp -d 192.168.203.18@tcp -m GET -e local_error -r 1 [ 1026.415523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.118@tcp [ 1026.631691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1026.658670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1026.679091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1026.699663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1026.729184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1026.770821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1026.815335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1026.868238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1026.911993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1026.958331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1027.035907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1027.105721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1027.172597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1027.233297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1027.300644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1027.734940] LNet: 56920:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1027.746438] LNetError: 56920:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1027.777449] LNet: Removed LNI 192.168.203.18@tcp [ 1028.192718] Key type .llcrypt unregistered [ 1028.195026] Key type ._llcrypt unregistered [ 1045.407938] Key type ._llcrypt registered [ 1045.410165] Key type .llcrypt registered [ 1046.749661] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 05:49:02 (1783072142) [ 1047.460445] Key type .llcrypt unregistered [ 1047.462781] Key type ._llcrypt unregistered [ 1053.700814] Key type ._llcrypt registered [ 1053.703730] Key type .llcrypt registered [ 1053.742906] LNet: 57958:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1053.742906] mask usage: [+|-] ... [ 1053.892562] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1053.906621] alg: No test for adler32 (adler32-zlib) [ 1054.829757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1054.855831] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 1063.566586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.118@tcp [ 1076.855631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1076.932508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1076.972872] LNet: Added LNI 192.168.203.18@tcp1 [8/256/0/180] [ 1078.423106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.18@tcp -d 192.168.203.118@tcp -m GET -e local_interrupt -r 1 [ 1078.484905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.18@tcp -d 192.168.203.118@tcp1 -m GET -e local_interrupt -r 1 [ 1078.540164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.18@tcp -d 192.168.203.18@tcp -m GET -e local_interrupt -r 1 [ 1078.618630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.18@tcp -d 192.168.203.18@tcp1 -m GET -e local_interrupt -r 1 [ 1078.688832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.18@tcp1 -d 192.168.203.118@tcp -m GET -e local_interrupt -r 1 [ 1078.759497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.18@tcp1 -d 192.168.203.118@tcp1 -m GET -e local_interrupt -r 1 [ 1078.804445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.18@tcp1 -d 192.168.203.18@tcp -m GET -e local_interrupt -r 1 [ 1078.879861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.18@tcp1 -d 192.168.203.18@tcp1 -m GET -e local_interrupt -r 1 [ 1078.953754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.118@tcp [ 1078.990497] LNet: There was an unexpected network error while writing to 192.168.203.118: rc = -22 [ 1079.009040] LNet: 1 local NIs in recovery (showing 1): 192.168.203.18@tcp [ 1079.298635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1079.354846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1079.394892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1079.479291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1079.519844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1079.582311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1079.620880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1079.666472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1079.694689] LNet: 58387:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1079.727503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1079.781058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1079.849968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1079.899465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1079.954807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1079.997726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1080.051678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1080.690258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.18@tcp -d 192.168.203.118@tcp -m GET -e local_dropped -r 1 [ 1080.773393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.18@tcp -d 192.168.203.118@tcp1 -m GET -e local_dropped -r 1 [ 1080.836969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.18@tcp -d 192.168.203.18@tcp -m GET -e local_dropped -r 1 [ 1080.899368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.18@tcp -d 192.168.203.18@tcp1 -m GET -e local_dropped -r 1 [ 1080.946207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.18@tcp1 -d 192.168.203.118@tcp -m GET -e local_dropped -r 1 [ 1081.021258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.18@tcp1 -d 192.168.203.118@tcp1 -m GET -e local_dropped -r 1 [ 1081.088550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.18@tcp1 -d 192.168.203.18@tcp -m GET -e local_dropped -r 1 [ 1081.161305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.18@tcp1 -d 192.168.203.18@tcp1 -m GET -e local_dropped -r 1 [ 1081.236498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.118@tcp [ 1081.273531] LNet: There was an unexpected network error while writing to 192.168.203.118: rc = -22 [ 1081.287415] LNet: Skipped 2 previous similar messages [ 1081.604504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1081.632769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1081.683450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1081.720787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1081.770643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1081.809410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1081.866828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1081.930592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1081.956485] LNet: 58509:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1081.992554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1082.051369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1082.113188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1082.162347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1082.197421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1082.232106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1082.270506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1082.832936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.18@tcp -d 192.168.203.118@tcp -m GET -e local_aborted -r 1 [ 1082.886913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.18@tcp -d 192.168.203.118@tcp1 -m GET -e local_aborted -r 1 [ 1082.938815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.18@tcp -d 192.168.203.18@tcp -m GET -e local_aborted -r 1 [ 1083.007109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.18@tcp -d 192.168.203.18@tcp1 -m GET -e local_aborted -r 1 [ 1083.060785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.18@tcp1 -d 192.168.203.118@tcp -m GET -e local_aborted -r 1 [ 1083.121265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.18@tcp1 -d 192.168.203.118@tcp1 -m GET -e local_aborted -r 1 [ 1083.188418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.18@tcp1 -d 192.168.203.18@tcp -m GET -e local_aborted -r 1 [ 1083.255672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.18@tcp1 -d 192.168.203.18@tcp1 -m GET -e local_aborted -r 1 [ 1083.312595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.118@tcp [ 1083.322348] LNet: There was an unexpected network error while writing to 192.168.203.118: rc = -22 [ 1083.338333] LNet: Skipped 2 previous similar messages [ 1083.720470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1083.782841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1083.808988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1083.837480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1083.867427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1083.899378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1083.975386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1084.028222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1084.044416] LNet: 58630:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1084.056416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1084.079360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1084.104375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1084.142762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1084.182885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1084.217996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1084.253576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1084.912335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.18@tcp -d 192.168.203.118@tcp -m GET -e local_no_route -r 1 [ 1084.970586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.18@tcp -d 192.168.203.118@tcp1 -m GET -e local_no_route -r 1 [ 1085.027449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.18@tcp -d 192.168.203.18@tcp -m GET -e local_no_route -r 1 [ 1085.076312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.18@tcp -d 192.168.203.18@tcp1 -m GET -e local_no_route -r 1 [ 1085.116477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.18@tcp1 -d 192.168.203.118@tcp -m GET -e local_no_route -r 1 [ 1085.148302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.18@tcp1 -d 192.168.203.118@tcp1 -m GET -e local_no_route -r 1 [ 1085.184475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.18@tcp1 -d 192.168.203.18@tcp -m GET -e local_no_route -r 1 [ 1085.224640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.18@tcp1 -d 192.168.203.18@tcp1 -m GET -e local_no_route -r 1 [ 1085.281374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.118@tcp [ 1085.555636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1085.604174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1085.631514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1085.671435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1085.702326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1085.750566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1085.786370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1085.814517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1085.868918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1085.906986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1085.933586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1085.977111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1086.024263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1086.068595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1086.111197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1086.831478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.18@tcp -d 192.168.203.118@tcp -m GET -e local_timeout -r 1 [ 1086.899432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.18@tcp -d 192.168.203.118@tcp1 -m GET -e local_timeout -r 1 [ 1086.985705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.18@tcp -d 192.168.203.18@tcp -m GET -e local_timeout -r 1 [ 1087.049815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.18@tcp -d 192.168.203.18@tcp1 -m GET -e local_timeout -r 1 [ 1087.114700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.18@tcp1 -d 192.168.203.118@tcp -m GET -e local_timeout -r 1 [ 1087.155815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.18@tcp1 -d 192.168.203.118@tcp1 -m GET -e local_timeout -r 1 [ 1087.226327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.18@tcp1 -d 192.168.203.18@tcp -m GET -e local_timeout -r 1 [ 1087.284542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.18@tcp1 -d 192.168.203.18@tcp1 -m GET -e local_timeout -r 1 [ 1087.349611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.118@tcp [ 1087.384079] LNet: There was an unexpected network error while writing to 192.168.203.118: rc = -22 [ 1087.387341] LNet: Skipped 5 previous similar messages [ 1087.690076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1087.753554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1087.815672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1087.847450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1087.880323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1087.914840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1087.945833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1087.973811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1087.988899] LNet: 58873:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1088.005483] LNet: 58873:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1088.030429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1088.074041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1088.110912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1088.155220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1088.206747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1088.250571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1088.284798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1088.845365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.18@tcp -d 192.168.203.118@tcp -m GET -e local_error -r 1 [ 1088.881119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.18@tcp -d 192.168.203.118@tcp1 -m GET -e local_error -r 1 [ 1088.926040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.18@tcp -d 192.168.203.18@tcp -m GET -e local_error -r 1 [ 1088.980432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.18@tcp -d 192.168.203.18@tcp1 -m GET -e local_error -r 1 [ 1089.079741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.18@tcp1 -d 192.168.203.118@tcp -m GET -e local_error -r 1 [ 1089.162471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.18@tcp1 -d 192.168.203.118@tcp1 -m GET -e local_error -r 1 [ 1089.240668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.18@tcp1 -d 192.168.203.18@tcp -m GET -e local_error -r 1 [ 1089.295412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.18@tcp1 -d 192.168.203.18@tcp1 -m GET -e local_error -r 1 [ 1089.381724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.118@tcp [ 1089.726105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1089.773089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1089.814072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1089.868637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1089.940223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1089.986813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1090.030788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1090.064485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1090.114284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1090.169488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1090.200742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1090.239455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1090.279749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1090.350323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1090.386460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1096.238807] LNet: 59070:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1096.245832] LNetError: 59070:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1097.311166] LNet: Removed LNI 192.168.203.18@tcp [ 1097.312648] LNetError: 59070:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1097.332151] LNet: Removed LNI 192.168.203.18@tcp1 [ 1097.559528] Key type .llcrypt unregistered [ 1097.561680] Key type ._llcrypt unregistered [ 1115.075656] Key type ._llcrypt registered [ 1115.077639] Key type .llcrypt registered [ 1117.024727] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 05:50:11 (1783072211) [ 1117.894587] Key type .llcrypt unregistered [ 1117.898126] Key type ._llcrypt unregistered [ 1125.483796] Key type ._llcrypt registered [ 1125.485381] Key type .llcrypt registered [ 1125.506711] LNet: 60107:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1125.506711] mask usage: [+|-] ... [ 1125.729115] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1125.756027] alg: No test for adler32 (adler32-zlib) [ 1126.842596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1126.882157] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 1135.960292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.118@tcp [ 1137.037606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.18@tcp -d 192.168.203.118@tcp -m GET -e remote_dropped -r 1 [ 1137.093820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.18@tcp -d 192.168.203.18@tcp -m GET -e remote_dropped -r 1 [ 1137.142645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.118@tcp [ 1137.168555] LNet: There was an unexpected network error while writing to 192.168.203.118: rc = -22 [ 1137.517457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1137.568527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1137.622370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1137.677623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1137.752398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1137.825468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1137.891407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1137.978781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1138.008841] LNet: 60322:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1138.073378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1138.110283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1138.165598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1138.221965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1138.327386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1138.407466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1138.469065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1139.479649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.18@tcp -d 192.168.203.118@tcp -m GET -e remote_error -r 1 [ 1139.570670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.18@tcp -d 192.168.203.18@tcp -m GET -e remote_error -r 1 [ 1139.640396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.118@tcp [ 1139.686732] LNet: There was an unexpected network error while writing to 192.168.203.118: rc = -22 [ 1140.129259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1140.154510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1140.183348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1140.218433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1140.243481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1140.285888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1140.320621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1140.357743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1140.390964] LNet: 60432:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1140.412995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1140.494434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1140.544116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1140.653726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1140.708101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1140.791877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1140.863227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1141.770919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.18@tcp -d 192.168.203.118@tcp -m GET -e remote_timeout -r 1 [ 1141.850880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.18@tcp -d 192.168.203.18@tcp -m GET -e remote_timeout -r 1 [ 1141.947340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.118@tcp [ 1142.022549] LNet: There was an unexpected network error while writing to 192.168.203.118: rc = -22 [ 1142.443914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1142.496226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1142.544926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1142.609625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1142.649446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1142.680811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1142.714315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1142.751501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1142.771789] LNet: 60541:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1142.790408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1142.831690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1142.865698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1142.922559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1142.996768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1143.075499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1143.151818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1143.735567] LNet: 60582:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1143.760338] LNetError: 60582:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1143.803725] LNet: Removed LNI 192.168.203.18@tcp [ 1144.481357] Key type .llcrypt unregistered [ 1144.483334] Key type ._llcrypt unregistered [ 1164.203899] Key type ._llcrypt registered [ 1164.209797] Key type .llcrypt registered [ 1165.792313] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 05:51:00 (1783072260) [ 1166.677345] Key type .llcrypt unregistered [ 1166.680296] Key type ._llcrypt unregistered [ 1172.428293] Key type ._llcrypt registered [ 1172.434294] Key type .llcrypt registered [ 1172.471247] LNet: 61620:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1172.471247] mask usage: [+|-] ... [ 1172.624931] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1172.641979] alg: No test for adler32 (adler32-zlib) [ 1173.550604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1173.583492] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 1180.302692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.118@tcp [ 1188.857553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1188.896746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1188.928469] LNet: Added LNI 192.168.203.18@tcp1 [8/256/0/180] [ 1190.294738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.18@tcp -d 192.168.203.118@tcp -m GET -e remote_dropped -r 1 [ 1190.328445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.18@tcp -d 192.168.203.118@tcp1 -m GET -e remote_dropped -r 1 [ 1190.358417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.18@tcp -d 192.168.203.18@tcp -m GET -e remote_dropped -r 1 [ 1190.388807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.18@tcp -d 192.168.203.18@tcp1 -m GET -e remote_dropped -r 1 [ 1190.415717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.18@tcp1 -d 192.168.203.118@tcp -m GET -e remote_dropped -r 1 [ 1190.446835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.18@tcp1 -d 192.168.203.118@tcp1 -m GET -e remote_dropped -r 1 [ 1190.475417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.18@tcp1 -d 192.168.203.18@tcp -m GET -e remote_dropped -r 1 [ 1190.506918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.18@tcp1 -d 192.168.203.18@tcp1 -m GET -e remote_dropped -r 1 [ 1190.534801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.118@tcp [ 1190.551662] LNet: There was an unexpected network error while writing to 192.168.203.118: rc = -22 [ 1190.559735] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.118@tcp [ 1190.752809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1190.777845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1190.798852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1190.821861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1190.860650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1190.893665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1190.919782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1190.959464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1190.980317] LNet: 62048:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1190.994190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1191.051268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1191.089644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1191.117579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1191.153335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1191.188909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1191.216346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1191.270321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1191.740356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.18@tcp -d 192.168.203.118@tcp -m GET -e remote_error -r 1 [ 1191.773692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.18@tcp -d 192.168.203.118@tcp1 -m GET -e remote_error -r 1 [ 1191.829255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.18@tcp -d 192.168.203.18@tcp -m GET -e remote_error -r 1 [ 1191.872121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.18@tcp -d 192.168.203.18@tcp1 -m GET -e remote_error -r 1 [ 1191.899861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.18@tcp1 -d 192.168.203.118@tcp -m GET -e remote_error -r 1 [ 1191.953408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.18@tcp1 -d 192.168.203.118@tcp1 -m GET -e remote_error -r 1 [ 1191.998953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.18@tcp1 -d 192.168.203.18@tcp -m GET -e remote_error -r 1 [ 1192.055981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.18@tcp1 -d 192.168.203.18@tcp1 -m GET -e remote_error -r 1 [ 1192.099041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.118@tcp [ 1192.135921] LNet: There was an unexpected network error while writing to 192.168.203.118: rc = -22 [ 1192.146699] LNet: Skipped 2 previous similar messages [ 1192.409750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1192.442487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1192.476584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1192.509770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1192.539848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1192.576227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1192.626970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1192.678201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1192.709527] LNet: 62171:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1192.730421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1192.755853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1192.801338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1192.833392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1192.861706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1192.895113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1192.926914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1192.982660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1193.545681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.18@tcp -d 192.168.203.118@tcp -m GET -e remote_timeout -r 1 [ 1193.581292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.18@tcp -d 192.168.203.118@tcp1 -m GET -e remote_timeout -r 1 [ 1193.633509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.18@tcp -d 192.168.203.18@tcp -m GET -e remote_timeout -r 1 [ 1193.685423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.18@tcp -d 192.168.203.18@tcp1 -m GET -e remote_timeout -r 1 [ 1193.715584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.18@tcp1 -d 192.168.203.118@tcp -m GET -e remote_timeout -r 1 [ 1193.756895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.18@tcp1 -d 192.168.203.118@tcp1 -m GET -e remote_timeout -r 1 [ 1193.787790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.18@tcp1 -d 192.168.203.18@tcp -m GET -e remote_timeout -r 1 [ 1193.832239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.18@tcp1 -d 192.168.203.18@tcp1 -m GET -e remote_timeout -r 1 [ 1193.870955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.118@tcp [ 1193.893413] LNet: There was an unexpected network error while writing to 192.168.203.118: rc = -22 [ 1194.114867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1194.143667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1194.184391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1194.205803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1194.231399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1194.254908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1194.278974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1194.302766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1194.320545] LNet: 62295:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1194.337365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1194.360586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1194.381169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1194.409251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1194.438823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1194.483191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1194.520734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1194.564893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1199.550744] LNet: 62372:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1199.564589] LNetError: 62372:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1199.580048] LNet: Removed LNI 192.168.203.18@tcp [ 1199.617214] LNet: Removed LNI 192.168.203.18@tcp1 [ 1200.028689] Key type .llcrypt unregistered [ 1200.033542] Key type ._llcrypt unregistered [ 1217.059389] Key type ._llcrypt registered [ 1217.063955] Key type .llcrypt registered [ 1218.393266] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 05:51:53 (1783072313) [ 1219.799666] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1221.704434] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 05:51:57 (1783072317) [ 1222.419829] Key type .llcrypt unregistered [ 1222.423819] Key type ._llcrypt unregistered [ 1228.352712] Key type ._llcrypt registered [ 1228.357365] Key type .llcrypt registered [ 1228.385685] LNet: 63701:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1228.385685] mask usage: [+|-] ... [ 1228.503688] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1228.521344] alg: No test for adler32 (adler32-zlib) [ 1229.433911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1229.453942] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 1236.580505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.118@tcp [ 1237.237424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.18@tcp -d 192.168.203.118@tcp -m GET -e network_timeout -r 1 [ 1237.270378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.18@tcp -d 192.168.203.18@tcp -m GET -e network_timeout -r 1 [ 1237.307574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.118@tcp [ 1237.322591] LNet: There was an unexpected network error while writing to 192.168.203.118: rc = -22 [ 1237.512662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1237.542449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1237.575571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1237.627718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1237.659806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1237.663212] LNet: 1 local NIs in recovery (showing 1): 192.168.203.18@tcp [ 1237.688132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1237.722338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1237.753662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1237.773369] LNet: 63915:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1237.786949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1237.822338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1237.859340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1237.903823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1237.953560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1238.006202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1238.233898] LNet: 63954:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1238.242704] LNetError: 63954:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1238.257723] LNet: Removed LNI 192.168.203.18@tcp [ 1238.717490] Key type .llcrypt unregistered [ 1238.723946] Key type ._llcrypt unregistered [ 1255.084851] Key type ._llcrypt registered [ 1255.088496] Key type .llcrypt registered [ 1255.128751] LNet: 64404:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1255.128751] mask usage: [+|-] ... [ 1255.259743] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1255.277383] alg: No test for adler32 (adler32-zlib) [ 1256.257766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1256.284514] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 1262.711446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.118@tcp [ 1270.410882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1270.493580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1270.528923] LNet: Added LNI 192.168.203.18@tcp1 [8/256/0/180] [ 1271.703285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.18@tcp -d 192.168.203.118@tcp -m GET -e network_timeout -r 1 [ 1271.754638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.18@tcp -d 192.168.203.118@tcp1 -m GET -e network_timeout -r 1 [ 1271.795930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.18@tcp -d 192.168.203.18@tcp -m GET -e network_timeout -r 1 [ 1271.855671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.18@tcp -d 192.168.203.18@tcp1 -m GET -e network_timeout -r 1 [ 1271.909547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.18@tcp1 -d 192.168.203.118@tcp -m GET -e network_timeout -r 1 [ 1271.962425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.18@tcp1 -d 192.168.203.118@tcp1 -m GET -e network_timeout -r 1 [ 1272.011653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.18@tcp1 -d 192.168.203.18@tcp -m GET -e network_timeout -r 1 [ 1272.053722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.18@tcp1 -d 192.168.203.18@tcp1 -m GET -e network_timeout -r 1 [ 1272.091741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.118@tcp [ 1272.111675] LNet: There was an unexpected network error while writing to 192.168.203.118: rc = -22 [ 1272.346200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1272.401703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1272.430600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1272.459192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1272.486487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1272.509454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1272.536915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1272.567544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1272.584942] LNet: 64832:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1272.598176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1272.621551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1272.652595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1272.671133] LNet: 1 local NIs in recovery (showing 1): 192.168.203.18@tcp [ 1272.684072] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.118@tcp [ 1272.689346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1272.729324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1272.758494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1276.839508] LNet: There was an unexpected network error while writing to 192.168.203.118: rc = -22 [ 1277.154801] LNet: 64906:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1277.162242] LNetError: 64906:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1277.174069] LNet: Removed LNI 192.168.203.18@tcp [ 1277.185712] LNet: Removed LNI 192.168.203.18@tcp1 [ 1277.577361] Key type .llcrypt unregistered [ 1277.579882] Key type ._llcrypt unregistered [ 1293.122400] Key type ._llcrypt registered [ 1293.127816] Key type .llcrypt registered [ 1294.364886] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 05:53:09 (1783072389) [ 1294.531451] LNet: 65704:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1294.531451] mask usage: [+|-] ... [ 1294.680586] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1294.692421] alg: No test for adler32 (adler32-zlib) [ 1295.663980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1295.729900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1295.775931] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 1296.523831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1296.557568] LNet: Added LNI 192.168.203.18@tcp1 [8/256/0/180] [ 1297.556157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.18@tcp [ 1297.639935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1297.680603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1297.713725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1297.733214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1297.762953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1297.808315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.18@tcp [ 1297.848778] LNetError: 66005:0:(api-ni.c:4400:lnet_ni_set_healthv()) manually adding local NI 192.168.203.18@tcp to recovery [ 1298.783394] LNet: 1 local NIs in recovery (showing 1): 192.168.203.18@tcp [ 1298.783719] LNet: There was an unexpected network error while writing to 192.168.203.18: rc = -22 [ 1300.831978] LNet: There was an unexpected network error while writing to 192.168.203.18: rc = -22 [ 1301.314598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1304.931339] LNet: There was an unexpected network error while writing to 192.168.203.18: rc = -22 [ 1306.203712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1306.273362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1306.414721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1306.465078] LNet: 66122:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1306.484949] LNetError: 66122:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1306.490813] LNet: Removed LNI 192.168.203.18@tcp [ 1306.511121] LNet: Removed LNI 192.168.203.18@tcp1 [ 1306.573081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1306.694399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1306.723834] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 1307.945341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1307.977433] LNet: Added LNI 192.168.203.18@tcp1 [8/256/0/180] [ 1309.092969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.18@tcp [ 1309.163320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1309.283023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1309.340164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1309.403212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1309.451365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1309.490927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1309.533628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.18@tcp [ 1309.554747] LNetError: 66395:0:(api-ni.c:4400:lnet_ni_set_healthv()) manually adding local NI 192.168.203.18@tcp to recovery [ 1309.728600] LNet: There was an unexpected network error while writing to 192.168.203.18: rc = -22 [ 1312.242175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1315.935387] LNet: There was an unexpected network error while writing to 192.168.203.18: rc = -22 [ 1315.949963] LNet: Skipped 1 previous similar message [ 1319.407915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1319.443450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1319.491553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1327.779222] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 05:53:43 (1783072423) [ 1328.037319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1328.088030] LNet: 67092:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1328.104345] LNetError: 67092:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1328.110763] LNetError: 67092:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1329.119174] LNet: Removed LNI 192.168.203.18@tcp [ 1329.126998] LNetError: 67092:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1329.149208] LNet: Removed LNI 192.168.203.18@tcp1 [ 1329.227650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1329.367667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1329.418879] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 1330.376330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1330.405382] LNet: Added LNI 192.168.203.18@tcp1 [8/256/0/180] [ 1331.541629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.18@tcp [ 1331.685836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1331.774926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1331.838530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1331.888427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1331.927586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1331.978370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.18@tcp --health 0 [ 1332.112794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1332.384299] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.18@tcp [ 1333.409231] LNet: There was an unexpected network error while writing to 192.168.203.18: rc = -22 [ 1333.421940] LNet: Skipped 1 previous similar message [ 1348.779623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1348.941433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1348.968661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1349.003986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.18@tcp --health 500 [ 1349.132965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1357.003989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1357.106701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1357.151970] LNet: 67600:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1357.159641] LNetError: 67600:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1357.169360] LNet: Removed LNI 192.168.203.18@tcp [ 1357.185364] LNet: Removed LNI 192.168.203.18@tcp1 [ 1357.205693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1357.249337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1357.268941] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 1358.252507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1358.277394] LNet: Added LNI 192.168.203.18@tcp1 [8/256/0/180] [ 1359.225492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.18@tcp [ 1359.275541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1359.364378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1359.404504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1359.464502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1359.517221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1359.567354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1359.604568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.18@tcp --health 0 [ 1360.293717] LNet: There was an unexpected network error while writing to 192.168.203.18: rc = -22 [ 1360.313396] LNet: Skipped 3 previous similar messages [ 1360.940417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1371.375736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1371.403638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1371.429325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1377.723540] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 05:54:33 (1783072473) [ 1378.215379] LNet: 68646:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1378.229447] LNetError: 68646:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1378.237744] LNetError: 68646:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1378.252467] LNet: Removed LNI 192.168.203.18@tcp [ 1378.261876] LNet: Removed LNI 192.168.203.18@tcp1 [ 1378.885849] Key type .llcrypt unregistered [ 1378.890755] Key type ._llcrypt unregistered [ 1386.728380] Key type ._llcrypt registered [ 1386.730362] Key type .llcrypt registered [ 1386.771108] LNet: 68869:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1386.771108] mask usage: [+|-] ... [ 1387.001222] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1387.011516] alg: No test for adler32 (adler32-zlib) [ 1387.973641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1388.033909] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 1397.366467] Lustre: DEBUG MARKER: Initial discovery [ 1397.494066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.118@tcp [ 1399.839568] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1399.906736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1399.962311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.118@tcp [ 1399.990616] LNet: There was an unexpected network error while writing to 192.168.203.118: rc = -22 [ 1400.029578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1400.076933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.203.118@tcp -r 1 -m GET -l 3 [ 1400.123471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.203.118@tcp -r 1 -m GET -e local_error [ 1400.179959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.203.118@tcp -r 1 -m PUT -l 6 [ 1400.352108] LNet: 1 local NIs in recovery (showing 1): 192.168.203.18@tcp [ 1401.428548] Lustre: DEBUG MARKER: Force oleg318-client.virtnet to discover 192.168.203.118@tcp (in background) [ 1401.676374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.118@tcp [ 1402.948712] Lustre: DEBUG MARKER: Force oleg318-server to discover 192.168.203.18@tcp [ 1404.767672] LNet: There was an unexpected network error while writing to 192.168.203.118: rc = -22 [ 1405.146260] Lustre: DEBUG MARKER: Wait for 69409 [ 1406.692323] Lustre: DEBUG MARKER: Finished wait on 69409 [ 1406.848371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1406.885255] LNet: There was an unexpected network error while writing to 192.168.203.118: rc = -22 [ 1406.922234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1407.209663] LNet: 69858:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1407.224528] LNetError: 69858:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1407.256068] LNet: Removed LNI 192.168.203.18@tcp [ 1407.722490] Key type .llcrypt unregistered [ 1407.731728] Key type ._llcrypt unregistered [ 1422.972142] Key type ._llcrypt registered [ 1422.976782] Key type .llcrypt registered [ 1424.187768] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 05:55:19 (1783072519) [ 1424.679288] Key type .llcrypt unregistered [ 1424.683885] Key type ._llcrypt unregistered [ 1430.343522] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1430.548399] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1430.558766] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1430.566174] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1433.039603] Key type ._llcrypt registered [ 1433.046963] Key type .llcrypt registered [ 1433.118830] LNet: 70900:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1433.118830] mask usage: [+|-] ... [ 1433.319604] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1433.343236] alg: No test for adler32 (adler32-zlib) [ 1434.274498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1434.369790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1434.406749] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 1435.257107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1435.297996] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1444.881734] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 05:55:40 (1783072540) [ 1445.036962] LNet: 71927:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1445.049806] LNetError: 71927:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1445.073602] LNet: Removed LNI 10.1.2.3@tcp [ 1445.095176] LNet: Removed LNI 192.168.203.18@tcp [ 1446.008359] Key type .llcrypt unregistered [ 1446.014229] Key type ._llcrypt unregistered [ 1455.060860] Key type ._llcrypt registered [ 1455.063546] Key type .llcrypt registered [ 1455.093341] LNet: 72184:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1455.093341] mask usage: [+|-] ... [ 1455.300751] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1455.318956] alg: No test for adler32 (adler32-zlib) [ 1456.259505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1456.303300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1456.324345] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 1457.116734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1457.141088] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1458.619479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.203.18@tcp [ 1459.743839] LNetError: Unexpected error -22 connecting to 192.168.203.18@tcp at host 192.168.203.18:988 [ 1464.558393] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 05:56:00 (1783072560) [ 1464.705121] LNet: 73065:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1464.717433] LNetError: 73065:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1464.729192] LNet: Removed LNI 10.1.2.3@tcp [ 1464.738204] LNet: Removed LNI 192.168.203.18@tcp [ 1465.301650] Key type .llcrypt unregistered [ 1465.304554] Key type ._llcrypt unregistered [ 1470.457866] Key type ._llcrypt registered [ 1470.461580] Key type .llcrypt registered [ 1470.495691] LNet: 73298:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1470.495691] mask usage: [+|-] ... [ 1470.702472] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1470.732230] alg: No test for adler32 (adler32-zlib) [ 1471.659948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1471.742493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1471.771050] LNet: Added LNI 192.168.203.18@tcp1 [8/256/0/180] [ 1472.560900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1472.593855] LNet: Added LNI 192.168.203.18@tcp2 [8/256/0/180] [ 1473.518339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.203.18@tcp1 --nid 192.168.203.18@tcp2 [ 1478.173317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.18@tcp1 192.168.203.18@tcp2 [ 1478.187505] LNetError: 73902:0:(lib-move.c:2007:lnet_handle_find_routed_path()) Unable to send message from 192.168.203.18@tcp1 to 192.168.203.18@tcp2 - Route table may be misconfigured [ 1478.209532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.18@tcp2 192.168.203.18@tcp1 [ 1484.543558] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 05:56:19 (1783072579) [ 1484.833287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1484.857757] LNet: 74474:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1484.886406] LNetError: 74474:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1485.926923] LNet: Removed LNI 192.168.203.18@tcp1 [ 1485.933458] LNetError: 74474:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1485.968103] LNet: Removed LNI 192.168.203.18@tcp2 [ 1485.990782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1486.037866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1486.064213] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 1486.796155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1486.828887] LNet: Added LNI 192.168.203.18@tcp1 [8/256/0/180] [ 1488.074232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.18@tcp [ 1488.116539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.18@tcp 192.168.203.18@tcp [ 1488.170422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.18@tcp1 192.168.203.18@tcp1 [ 1488.223665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.18@tcp -d 192.168.203.18@tcp -e network_timeout [ 1488.308442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.18@tcp -d 192.168.203.18@tcp1 -e network_timeout [ 1488.387973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.18@tcp1 -d 192.168.203.18@tcp -e network_timeout [ 1488.455981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.18@tcp1 -d 192.168.203.18@tcp1 -e network_timeout [ 1488.486190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.18@tcp [ 1488.501270] LNet: There was an unexpected network error while writing to 192.168.203.18: rc = -22 [ 1488.531624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1488.645349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1488.815840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1489.120095] LNet: 1 local NIs in recovery (showing 1): 192.168.203.18@tcp [ 1497.559906] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 05:56:32 (1783072592) [ 1498.025785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1498.070780] LNet: 75337:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1498.086456] LNetError: 75337:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1498.101417] LNet: Removed LNI 192.168.203.18@tcp [ 1499.127042] LNet: Removed LNI 192.168.203.18@tcp1 [ 1499.170381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1499.310200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1499.617470] LNet: 75374:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1500.182468] Key type .llcrypt unregistered [ 1500.185056] Key type ._llcrypt unregistered [ 1511.768158] Key type ._llcrypt registered [ 1511.770826] Key type .llcrypt registered [ 1513.353888] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 05:56:48 (1783072608) [ 1513.652358] LNet: 76155:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1513.652358] mask usage: [+|-] ... [ 1513.898258] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1513.906765] alg: No test for adler32 (adler32-zlib) [ 1514.818739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1516.123470] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1517.234631] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 05:56:53 (1783072613) [ 1517.461833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1517.482515] LNet: 76483:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1517.509687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1517.552168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1517.587023] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 1518.112346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1518.132776] LNet: Added LNI 192.168.203.18@tcp1 [8/256/0/180] [ 1518.714322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.18@tcp [ 1518.754478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.18@tcp1 [ 1518.790662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.18@tcp1 [ 1524.266521] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 05:56:59 (1783072619) [ 1525.676396] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg318-server [ 1527.105394] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 05:57:02 (1783072622) [ 1528.710953] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg318-server [ 1529.930317] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 05:57:05 (1783072625) [ 1531.390759] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg318-server [ 1533.024598] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 05:57:08 (1783072628) [ 1534.203650] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg318-server [ 1535.626340] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 05:57:11 (1783072631) [ 1537.143690] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg318-server [ 1538.364146] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 05:57:14 (1783072634) [ 1539.763971] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg318-server [ 1541.033196] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 05:57:16 (1783072636) [ 1542.558506] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg318-server [ 1543.994231] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 05:57:19 (1783072639) [ 1545.560095] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg318-server [ 1547.039359] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 05:57:22 (1783072642) [ 1548.571481] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg318-server [ 1549.901255] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 05:57:25 (1783072645) [ 1550.114590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1550.145125] LNet: 80358:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1550.155769] LNetError: 80358:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1550.161672] LNet: Removed LNI 192.168.203.18@tcp [ 1551.209616] LNet: Removed LNI 192.168.203.18@tcp1 [ 1551.242284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1551.326875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1551.355951] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 1551.913600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1551.971954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.18@tcp [ 1552.136950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1552.173586] LNet: 80505:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1552.178448] LNetError: 80505:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1552.184581] LNetError: 80505:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1553.252693] LNet: Removed LNI 192.168.203.18@tcp [ 1553.274791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1553.327772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1553.355836] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 1553.938439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.18@tcp [ 1554.093619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1554.113862] LNetError: 80651:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1554.124184] LNet: Removed LNI 192.168.203.18@tcp [ 1554.142326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1554.188442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1554.223390] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 1554.880062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1554.940939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.18@tcp [ 1555.116074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1555.142801] LNet: 80798:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1555.147899] LNet: 80798:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1556.200472] LNet: Removed LNI 192.168.203.18@tcp [ 1556.232182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1556.276959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1556.304147] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 1556.958176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.18@tcp [ 1557.145478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1557.165168] LNetError: 80943:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1557.173793] LNetError: 80943:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1557.186866] LNet: Removed LNI 192.168.203.18@tcp [ 1557.206805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1557.276322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1557.307467] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 1557.903266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1557.983439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.18@tcp [ 1558.162599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1559.212828] LNet: Removed LNI 192.168.203.18@tcp [ 1559.264381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1559.337868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1559.371476] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 1560.208978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.18@tcp [ 1560.442917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1560.466789] LNet: 81236:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1560.472109] LNet: 81236:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1561.514690] LNet: Removed LNI 192.168.203.18@tcp [ 1561.552908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1561.632394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1561.659760] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 1562.290815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1562.353742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.18@tcp [ 1562.549471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1562.567445] LNetError: 81383:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1562.573465] LNetError: 81383:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1562.599412] LNet: Removed LNI 192.168.203.18@tcp [ 1562.631866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1562.683350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1562.712666] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 1563.396359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.18@tcp [ 1563.592068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1564.645909] LNet: Removed LNI 192.168.203.18@tcp [ 1564.690149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1564.762382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1564.788122] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 1565.464405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1565.533291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.18@tcp [ 1565.752988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1566.821688] LNet: Removed LNI 192.168.203.18@tcp [ 1566.874168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1566.941381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1566.976768] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 1567.616876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.18@tcp [ 1567.828355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1568.934560] LNet: Removed LNI 192.168.203.18@tcp [ 1568.969105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1569.048475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1569.083618] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 1569.803949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1569.907669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.18@tcp [ 1570.151101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1570.175233] LNet: 81968:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1570.180724] LNet: 81968:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1571.237420] LNet: Removed LNI 192.168.203.18@tcp [ 1571.259445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1571.331993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1571.357743] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 1571.938989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.18@tcp [ 1574.398657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1574.419856] LNetError: 82132:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1574.425173] LNetError: 82132:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1574.434456] LNet: Removed LNI 192.168.203.18@tcp [ 1574.447229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1574.485384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1574.507664] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 1575.203813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1575.279544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1575.317612] LNet: Removed LNI 192.168.203.18@tcp [ 1575.335912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1575.388329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1575.427087] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 1576.205092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1581.170960] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 05:57:57 (1783072677) [ 1581.280142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1581.336357] LNet: Removed LNI 192.168.203.18@tcp [ 1581.355742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1581.400956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1581.425641] LNet: Added LNI 192.168.203.18@tcp231 [8/256/0/180] [ 1582.371844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1582.408362] LNet: Removed LNI 192.168.203.18@tcp231 [ 1582.417907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1582.457048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1582.477901] LNet: Added LNI 192.168.203.18@tcp231 [8/256/0/180] [ 1582.484234] LNet: 83114:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1582.824252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1582.858985] LNet: Removed LNI 192.168.203.18@tcp231 [ 1582.870337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1582.895443] LNet: Added LNI 192.168.203.18@tcp231 [8/256/0/180] [ 1583.639925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1583.671341] LNet: Removed LNI 192.168.203.18@tcp231 [ 1583.688235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1583.742608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1583.776298] LNet: Added LNI 192.168.203.18@tcp231 [8/256/0/0] [ 1583.786570] LNet: 83266:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1584.178950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1584.202589] LNet: Removed LNI 192.168.203.18@tcp231 [ 1584.222707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1584.255558] LNet: Added LNI 192.168.203.18@tcp231 [8/256/0/0] [ 1585.037403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1585.058633] LNet: Removed LNI 192.168.203.18@tcp231 [ 1585.069719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1585.125307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1585.147249] LNet: Added LNI 192.168.203.18@tcp231 [8/256/0/60] [ 1585.154716] LNet: 83417:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1585.425499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1585.448691] LNet: Removed LNI 192.168.203.18@tcp231 [ 1585.462819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1585.484736] LNet: Added LNI 192.168.203.18@tcp231 [8/256/0/60] [ 1586.263664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1586.284736] LNet: 83560:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1586.290266] LNet: 83560:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 6 previous similar messages [ 1586.308720] LNet: Removed LNI 192.168.203.18@tcp231 [ 1586.323303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1586.376020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1586.398807] LNet: Added LNI 192.168.203.18@tcp231 [8/256/0/180] [ 1586.650084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1586.678886] LNet: Removed LNI 192.168.203.18@tcp231 [ 1586.687415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1586.700501] LNet: Added LNI 192.168.203.18@tcp231 [8/256/0/180] [ 1591.629785] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 05:58:07 (1783072687) [ 1591.676886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1591.693218] LNetError: 84271:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1591.697770] LNetError: 84271:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1591.717109] LNet: Removed LNI 192.168.203.18@tcp231 [ 1591.728913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1591.792884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1591.825267] LNet: Added LNI 192.168.203.18@tcp232 [8/256/0/180] [ 1592.318862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1592.345291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1592.373313] LNet: Removed LNI 192.168.203.18@tcp232 [ 1592.392954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1592.427748] LNet: Added LNI 192.168.203.18@tcp232 [8/256/0/180] [ 1592.972521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1593.007306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1593.034549] LNet: Removed LNI 192.168.203.18@tcp232 [ 1593.054925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1593.078468] LNet: Added LNI 192.168.203.18@tcp232 [8/256/0/180] [ 1593.582597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1593.612991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1593.641996] LNet: Removed LNI 192.168.203.18@tcp232 [ 1593.661040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1593.680738] LNet: Added LNI 192.168.203.18@tcp232 [8/256/0/180] [ 1594.227483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1594.262843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1594.305473] LNet: Removed LNI 192.168.203.18@tcp232 [ 1594.314431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1598.596288] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 05:58:14 (1783072694) [ 1599.296783] Key type .llcrypt unregistered [ 1599.301147] Key type ._llcrypt unregistered [ 1603.487385] Key type ._llcrypt registered [ 1603.489672] Key type .llcrypt registered [ 1603.510311] LNet: 85634:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1603.510311] mask usage: [+|-] ... [ 1603.631259] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1603.642570] alg: No test for adler32 (adler32-zlib) [ 1604.492939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1604.513279] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 1609.449838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.118@tcp [ 1615.274434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1615.308103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1615.331690] LNet: Added LNI 192.168.203.18@tcp1 [8/256/0/180] [ 1616.366864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.18@tcp -d 192.168.203.118@tcp -m GET -e local_interrupt -r 2 [ 1616.396510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.18@tcp -d 192.168.203.118@tcp1 -m GET -e local_interrupt -r 2 [ 1616.422635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.18@tcp -d 192.168.203.18@tcp -m GET -e local_interrupt -r 2 [ 1616.451564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.18@tcp -d 192.168.203.18@tcp1 -m GET -e local_interrupt -r 2 [ 1616.478647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.18@tcp1 -d 192.168.203.118@tcp -m GET -e local_interrupt -r 2 [ 1616.507606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.18@tcp1 -d 192.168.203.118@tcp1 -m GET -e local_interrupt -r 2 [ 1616.537948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.18@tcp1 -d 192.168.203.18@tcp -m GET -e local_interrupt -r 2 [ 1616.580425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.18@tcp1 -d 192.168.203.18@tcp1 -m GET -e local_interrupt -r 2 [ 1616.638715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.18@tcp 192.168.203.118@tcp [ 1616.657876] LNet: There was an unexpected network error while writing to 192.168.203.118: rc = -22 [ 1616.667645] LNet: 1 local NIs in recovery (showing 1): 192.168.203.18@tcp [ 1616.865558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1616.897819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1616.926579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1616.952618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1616.977040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1617.004597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1617.027403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1617.048765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1617.064477] LNet: 86066:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1617.080146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1617.106558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1617.136657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1617.180150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1617.216613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1617.246179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1617.685823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.18@tcp -d 192.168.203.118@tcp -m GET -e local_dropped -r 2 [ 1617.719678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.18@tcp -d 192.168.203.118@tcp1 -m GET -e local_dropped -r 2 [ 1617.758081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.18@tcp -d 192.168.203.18@tcp -m GET -e local_dropped -r 2 [ 1617.800627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.18@tcp -d 192.168.203.18@tcp1 -m GET -e local_dropped -r 2 [ 1617.844259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.18@tcp1 -d 192.168.203.118@tcp -m GET -e local_dropped -r 2 [ 1617.879952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.18@tcp1 -d 192.168.203.118@tcp1 -m GET -e local_dropped -r 2 [ 1617.916279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.18@tcp1 -d 192.168.203.18@tcp -m GET -e local_dropped -r 2 [ 1617.939756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.18@tcp1 -d 192.168.203.18@tcp1 -m GET -e local_dropped -r 2 [ 1617.970533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.18@tcp 192.168.203.118@tcp [ 1617.982721] LNet: There was an unexpected network error while writing to 192.168.203.118: rc = -22 [ 1618.165656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1618.198604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1618.231924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1618.264077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1618.291888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1618.315672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1618.336464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1618.355491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1618.373927] LNet: 86186:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1618.391449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1618.417397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1618.450852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1618.481585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1618.512864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1618.539828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1619.023937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.18@tcp -d 192.168.203.118@tcp -m GET -e local_aborted -r 2 [ 1619.048684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.18@tcp -d 192.168.203.118@tcp1 -m GET -e local_aborted -r 2 [ 1619.077377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.18@tcp -d 192.168.203.18@tcp -m GET -e local_aborted -r 2 [ 1619.111733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.18@tcp -d 192.168.203.18@tcp1 -m GET -e local_aborted -r 2 [ 1619.142132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.18@tcp1 -d 192.168.203.118@tcp -m GET -e local_aborted -r 2 [ 1619.176190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.18@tcp1 -d 192.168.203.118@tcp1 -m GET -e local_aborted -r 2 [ 1619.205349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.18@tcp1 -d 192.168.203.18@tcp -m GET -e local_aborted -r 2 [ 1619.233726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.18@tcp1 -d 192.168.203.18@tcp1 -m GET -e local_aborted -r 2 [ 1619.274447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.18@tcp 192.168.203.118@tcp [ 1619.301365] LNet: There was an unexpected network error while writing to 192.168.203.118: rc = -22 [ 1619.480548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1619.511058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1619.537378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1619.565656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1619.595809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1619.619849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1619.646536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1619.678906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1619.701147] LNet: 86306:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1619.717297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1619.741000] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1619.765670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1619.789268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1619.833624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1619.868088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1620.341349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.18@tcp -d 192.168.203.118@tcp -m GET -e local_no_route -r 2 [ 1620.378083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.18@tcp -d 192.168.203.118@tcp1 -m GET -e local_no_route -r 2 [ 1620.414681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.18@tcp -d 192.168.203.18@tcp -m GET -e local_no_route -r 2 [ 1620.443329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.18@tcp -d 192.168.203.18@tcp1 -m GET -e local_no_route -r 2 [ 1620.469710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.18@tcp1 -d 192.168.203.118@tcp -m GET -e local_no_route -r 2 [ 1620.496451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.18@tcp1 -d 192.168.203.118@tcp1 -m GET -e local_no_route -r 2 [ 1620.528224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.18@tcp1 -d 192.168.203.18@tcp -m GET -e local_no_route -r 2 [ 1620.558960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.18@tcp1 -d 192.168.203.18@tcp1 -m GET -e local_no_route -r 2 [ 1620.593963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.18@tcp 192.168.203.118@tcp [ 1620.785161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1620.806765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1620.828150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1620.851426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1620.888660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1620.912744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1620.944814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1620.978784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1621.007372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1621.032904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1621.064455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1621.097891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1621.132961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1621.166593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1621.532581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.18@tcp -d 192.168.203.118@tcp -m GET -e local_timeout -r 2 [ 1621.577408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.18@tcp -d 192.168.203.118@tcp1 -m GET -e local_timeout -r 2 [ 1621.613286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.18@tcp -d 192.168.203.18@tcp -m GET -e local_timeout -r 2 [ 1621.653173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.18@tcp -d 192.168.203.18@tcp1 -m GET -e local_timeout -r 2 [ 1621.676348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.18@tcp1 -d 192.168.203.118@tcp -m GET -e local_timeout -r 2 [ 1621.695784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.18@tcp1 -d 192.168.203.118@tcp1 -m GET -e local_timeout -r 2 [ 1621.718864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.18@tcp1 -d 192.168.203.18@tcp -m GET -e local_timeout -r 2 [ 1621.742558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.18@tcp1 -d 192.168.203.18@tcp1 -m GET -e local_timeout -r 2 [ 1621.775817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.18@tcp 192.168.203.118@tcp [ 1621.790981] LNet: There was an unexpected network error while writing to 192.168.203.118: rc = -22 [ 1621.795336] LNet: Skipped 1 previous similar message [ 1621.922413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1621.944506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1621.970084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1621.994624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1622.017917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1622.039486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1622.066269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1622.089662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1622.104546] LNet: 86546:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1622.110275] LNet: 86546:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1622.122273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1622.153706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1622.177884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1622.202481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1622.228720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1622.261788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1622.680912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.18@tcp -d 192.168.203.118@tcp -m GET -e remote_dropped -r 2 [ 1622.702896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.18@tcp -d 192.168.203.118@tcp1 -m GET -e remote_dropped -r 2 [ 1622.729745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.18@tcp -d 192.168.203.18@tcp -m GET -e remote_dropped -r 2 [ 1622.756787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.18@tcp -d 192.168.203.18@tcp1 -m GET -e remote_dropped -r 2 [ 1622.788671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.18@tcp1 -d 192.168.203.118@tcp -m GET -e remote_dropped -r 2 [ 1622.823941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.18@tcp1 -d 192.168.203.118@tcp1 -m GET -e remote_dropped -r 2 [ 1622.855294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.18@tcp1 -d 192.168.203.18@tcp -m GET -e remote_dropped -r 2 [ 1622.884784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.18@tcp1 -d 192.168.203.18@tcp1 -m GET -e remote_dropped -r 2 [ 1622.913590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.18@tcp 192.168.203.118@tcp [ 1622.931274] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.118@tcp [ 1623.149527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1623.168098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1623.191611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1623.219917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1623.256068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1623.283710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1623.307482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1623.331798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1623.368022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1623.397927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1623.437709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1623.472399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1623.504776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1623.531508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1627.271760] LNet: 86740:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1627.279545] LNetError: 86740:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1628.319807] LNet: Removed LNI 192.168.203.18@tcp [ 1628.328519] LNetError: 86740:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1628.339289] LNet: Removed LNI 192.168.203.18@tcp1 [ 1628.638819] Key type .llcrypt unregistered [ 1628.642844] Key type ._llcrypt unregistered [ 1640.310052] Key type ._llcrypt registered [ 1640.313538] Key type .llcrypt registered [ 1641.564355] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 05:58:56 (1783072736) [ 1641.777886] LNet: 87545:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1641.777886] mask usage: [+|-] ... [ 1641.913351] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1641.940219] alg: No test for adler32 (adler32-zlib) [ 1642.859046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1642.913426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1642.935302] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 1643.583662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1643.605723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.18@tcp [ 1643.621641] LNetError: 87718:0:(api-ni.c:4400:lnet_ni_set_healthv()) manually adding local NI 192.168.203.18@tcp to recovery [ 1643.690171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.203.18@tcp [ 1643.761682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1648.896926] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 05:59:04 (1783072744) [ 1648.958534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1648.977961] LNet: 88300:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1648.983858] LNetError: 88300:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1649.002231] LNet: Removed LNI 192.168.203.18@tcp [ 1649.017285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1649.080871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1649.113302] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 1649.644127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1649.669859] LNet: Added LNI 192.168.203.18@tcp1 [8/256/0/180] [ 1650.446148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.203.18@tcp [ 1650.461796] LNetError: 88573:0:(api-ni.c:4400:lnet_ni_set_healthv()) manually adding local NI 192.168.203.18@tcp to recovery [ 1651.104885] LNet: 1 local NIs in recovery (showing 1): 192.168.203.18@tcp [ 1651.105783] LNet: There was an unexpected network error while writing to 192.168.203.18: rc = -22 [ 1651.627446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.203.18@tcp [ 1658.409795] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 05:59:14 (1783072754) [ 1659.968516] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg318-server [ 1661.766795] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 05:59:17 (1783072757) [ 1663.523781] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg318-server [ 1665.017165] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 05:59:20 (1783072760) [ 1666.407804] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg318-server [ 1667.897352] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 05:59:23 (1783072763) [ 1668.053128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1668.080846] LNet: 90203:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1668.086336] LNetError: 90203:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1668.092798] LNet: Removed LNI 192.168.203.18@tcp [ 1668.111351] LNet: Removed LNI 192.168.203.18@tcp1 [ 1668.133406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1668.181971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1668.202798] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 1668.744847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1668.795406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1668.849593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1668.925441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1668.973020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1669.030150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1674.014137] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 05:59:29 (1783072769) [ 1674.075472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1674.096158] LNet: 90922:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1674.108404] LNetError: 90922:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1674.114968] LNetError: 90922:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1674.129858] LNet: Removed LNI 192.168.203.18@tcp [ 1674.146216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1674.196253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1674.211760] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 1679.548466] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 05:59:35 (1783072775) [ 1679.721451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1679.742439] LNet: 91630:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1679.748641] LNetError: 91630:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1679.761496] LNet: Removed LNI 192.168.203.18@tcp [ 1679.785020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1679.835685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1679.859768] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 1684.521955] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 05:59:40 (1783072780) [ 1685.559836] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1686.620536] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 05:59:42 (1783072782) [ 1687.082905] LNet: 92665:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1687.090685] LNetError: 92665:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1687.106102] LNet: Removed LNI 192.168.203.18@tcp [ 1687.564800] Key type .llcrypt unregistered [ 1687.567704] Key type ._llcrypt unregistered [ 1691.718253] Key type ._llcrypt registered [ 1691.721702] Key type .llcrypt registered [ 1691.763863] LNet: 92894:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1691.763863] mask usage: [+|-] ... [ 1691.903587] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1691.917486] alg: No test for adler32 (adler32-zlib) [ 1692.748853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1692.771632] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 1698.255264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.118@tcp [ 1702.022617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.203.118@tcp [ 1702.292050] LNet: 93081:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1702.297067] LNetError: 93081:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1702.314479] LNet: Removed LNI 192.168.203.18@tcp [ 1702.627218] Key type .llcrypt unregistered [ 1702.630552] Key type ._llcrypt unregistered [ 1710.758457] Key type ._llcrypt registered [ 1710.760917] Key type .llcrypt registered [ 1711.930648] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1713.095612] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1714.297843] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 06:00:10 (1783072810) [ 1715.568257] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1716.980577] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 06:00:12 (1783072812) [ 1718.308653] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg318-server [ 1719.599715] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 06:00:15 (1783072815) [ 1720.994451] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg318-server [ 1722.165563] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 06:00:17 (1783072817) [ 1722.976353] LNet: 95134:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1722.976353] mask usage: [+|-] ... [ 1723.261251] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1723.274744] alg: No test for adler32 (adler32-zlib) [ 1724.305376] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 1724.492835] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 1726.135128] Key type lgssc registered [ 1726.998953] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1741.362900] Key type lgssc unregistered [ 1741.542203] LNet: 96255:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1741.549802] LNetError: 96255:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1741.561991] LNet: Removed LNI 192.168.203.18@tcp [ 1742.036062] Key type .llcrypt unregistered [ 1742.037806] Key type ._llcrypt unregistered [ 1752.064277] Key type ._llcrypt registered [ 1752.066054] Key type .llcrypt registered [ 1752.967069] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 06:00:48 (1783072848) [ 1753.155984] LNet: 97051:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1753.155984] mask usage: [+|-] ... [ 1753.262900] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1753.280180] alg: No test for adler32 (adler32-zlib) [ 1754.128938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1754.172908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1754.203664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1754.238696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1754.265652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1758.807182] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 06:00:54 (1783072854) [ 1758.878034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1758.896496] LNet: 97663:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1758.910845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1758.950627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1758.968621] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 1759.737258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1759.751648] LNet: 97813:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1759.756609] LNetError: 97813:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1759.769623] LNet: Removed LNI 192.168.203.18@tcp [ 1759.784991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1759.843580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1759.887534] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 1760.334753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1760.366578] LNetError: 97841:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1760.383116] LNet: Removed LNI 192.168.203.18@tcp [ 1760.399454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1760.438418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1760.471350] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 1765.262454] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 06:01:00 (1783072860) [ 1765.339444] LNet: 98438:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1765.346542] LNet: 98438:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1765.356106] LNetError: 98438:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1765.380350] LNet: Removed LNI 192.168.203.18@tcp [ 1765.888386] Key type .llcrypt unregistered [ 1765.898341] Key type ._llcrypt unregistered [ 1769.965530] Key type ._llcrypt registered [ 1769.968617] Key type .llcrypt registered [ 1769.995868] LNet: 98675:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1769.995868] mask usage: [+|-] ... [ 1770.174269] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1770.191780] alg: No test for adler32 (adler32-zlib) [ 1771.049723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1771.143227] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 1771.145594] LNet: *** cfs_fail_loc=e003, val=100*** [ 1771.696786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.18@tcp [ 1771.718584] LNet: *** cfs_fail_loc=e003, val=100*** [ 1771.722207] LNet: Skipped 2 previous similar messages [ 1771.726732] Ping reply received with mapping: [ 1771.729400] found metadata reply NI[192.168.203.18@tcp] [ 1771.731878] 12 cb a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1771.734391] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1771.736507] 5a 5a 5a 5a 5a 5a 5a 5a [ 1771.830645] LNet: 98842:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1771.836479] LNetError: 98842:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1771.847934] LNet: Removed LNI 192.168.203.18@tcp [ 1772.299319] Key type .llcrypt unregistered [ 1772.301317] Key type ._llcrypt unregistered [ 1779.357218] Key type ._llcrypt registered [ 1779.358869] Key type .llcrypt registered [ 1780.296755] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 06:01:16 (1783072876) [ 1780.591300] Key type .llcrypt unregistered [ 1780.594556] Key type ._llcrypt unregistered [ 1784.811058] Key type ._llcrypt registered [ 1784.814023] Key type .llcrypt registered [ 1784.853054] LNet: 99835:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1784.853054] mask usage: [+|-] ... [ 1785.015915] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1785.036958] alg: No test for adler32 (adler32-zlib) [ 1786.030405] LNetError: 99886:0:(api-ni.c:2779:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1786.332411] Key type .llcrypt unregistered [ 1786.336593] Key type ._llcrypt unregistered [ 1789.793299] Key type ._llcrypt registered [ 1789.794915] Key type .llcrypt registered [ 1790.719711] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 06:01:26 (1783072886) [ 1791.030442] Key type .llcrypt unregistered [ 1791.040458] Key type ._llcrypt unregistered [ 1794.916405] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1794.920853] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1797.273590] Key type ._llcrypt registered [ 1797.275378] Key type .llcrypt registered [ 1797.296329] LNet: 100752:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1797.296329] mask usage: [+|-] ... [ 1797.380492] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1797.392080] alg: No test for adler32 (adler32-zlib) [ 1798.254241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1798.312439] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1805.280955] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 06:01:41 (1783072901) [ 1805.361306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1805.384841] LNet: 101661:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1805.388394] LNetError: 101661:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1805.407694] LNet: Removed LNI 10.1.2.3@tcp [ 1805.425593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1805.912605] Key type .llcrypt unregistered [ 1805.916140] Key type ._llcrypt unregistered [ 1810.080534] Key type ._llcrypt registered [ 1810.083317] Key type .llcrypt registered [ 1810.121798] LNet: 101896:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1810.121798] mask usage: [+|-] ... [ 1810.249819] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1810.265645] alg: No test for adler32 (adler32-zlib) [ 1811.234710] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 1811.306280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.18@tcp [ 1811.442586] LNet: 101954:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1811.447734] LNetError: 101954:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1811.459195] LNet: Removed LNI 192.168.203.18@tcp [ 1811.914546] Key type .llcrypt unregistered [ 1811.918576] Key type ._llcrypt unregistered [ 1819.559775] Key type ._llcrypt registered [ 1819.562482] Key type .llcrypt registered [ 1820.470234] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 06:01:56 (1783072916) [ 1821.034258] Key type .llcrypt unregistered [ 1821.036847] Key type ._llcrypt unregistered [ 1825.417923] Key type ._llcrypt registered [ 1825.421640] Key type .llcrypt registered [ 1825.461380] LNet: 102937:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1825.461380] mask usage: [+|-] ... [ 1825.596640] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1825.614220] alg: No test for adler32 (adler32-zlib) [ 1830.968240] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 06:02:06 (1783072926) [ 1831.069621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1831.124486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1831.175322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1831.191372] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 1831.802686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1831.848558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1831.878408] LNet: Removed LNI 192.168.203.18@tcp [ 1832.031232] LNet: 103696:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1832.410313] Key type .llcrypt unregistered [ 1832.413871] Key type ._llcrypt unregistered [ 1840.108973] Key type ._llcrypt registered [ 1840.111415] Key type .llcrypt registered [ 1840.996346] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 06:02:16 (1783072936) [ 1841.137804] LNet: 104460:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1841.137804] mask usage: [+|-] ... [ 1841.257534] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1841.270992] alg: No test for adler32 (adler32-zlib) [ 1842.114238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1842.146831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1842.165891] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 1842.721327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.18@tcp [ 1842.759740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.203.18@tcp [ 1842.775684] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1842.779259] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1842.782164] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1842, 0, 0 [ 1842.787404] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1842.789771] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1842.793773] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1842.797819] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1842, 0, 0 [ 1842.802147] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1842.804620] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1842.807932] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1842.810968] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1842, 0, 1892 [ 1842.815761] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1842.818459] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1842.823195] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1842.826311] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1842, 0, 1892 [ 1842.832831] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1846.661458] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 06:02:22 (1783072942) [ 1846.966401] LNet: 105234:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1846.976436] LNetError: 105234:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1846.992776] LNet: Removed LNI 192.168.203.18@tcp [ 1847.395544] Key type .llcrypt unregistered [ 1847.399968] Key type ._llcrypt unregistered [ 1851.036515] Key type ._llcrypt registered [ 1851.040079] Key type .llcrypt registered [ 1851.075605] LNet: 105457:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1851.075605] mask usage: [+|-] ... [ 1851.226709] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1851.239943] alg: No test for adler32 (adler32-zlib) [ 1852.096635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1852.111910] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 1856.987839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.118@tcp [ 1862.449639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1862.477470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1862.496376] LNet: Added LNI 192.168.203.18@tcp1 [8/256/0/180] [ 1863.146150] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1863.205241] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1863.208503] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1863.212573] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1865.491241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1865.507824] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1872.383752] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 06:02:48 (1783072968) [ 1872.536511] LNet: 106739:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1872.545044] LNetError: 106739:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1872.553778] LNet: Removed LNI 192.168.203.18@tcp [ 1873.567272] LNet: Removed LNI 192.168.203.18@tcp1 [ 1873.571541] LNetError: 106739:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1873.580223] LNetError: 106739:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1873.600789] LNet: Removed LNI 10.1.2.3@tcp99 [ 1874.063650] Key type .llcrypt unregistered [ 1874.065341] Key type ._llcrypt unregistered [ 1878.272440] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1878.276714] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1880.652175] Key type ._llcrypt registered [ 1880.654353] Key type .llcrypt registered [ 1880.675975] LNet: 107004:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1880.675975] mask usage: [+|-] ... [ 1880.790727] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1880.804775] alg: No test for adler32 (adler32-zlib) [ 1881.673026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1881.717445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1881.740514] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 1882.230032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1882.261954] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1883.671765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 1883.709067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1883.740465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 1883.755306] LNetError: 107482:0:(peer.c:2234:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1883.783487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1883.833195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.18@tcp [ 1884.196507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1884.244726] LNet: 107502:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1884.253740] LNetError: 107502:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1884.258744] LNet: Removed LNI 10.1.2.3@tcp [ 1885.293068] LNet: Removed LNI 192.168.203.18@tcp [ 1885.726287] Key type .llcrypt unregistered [ 1885.731419] Key type ._llcrypt unregistered [ 1893.612382] Key type ._llcrypt registered [ 1893.617255] Key type .llcrypt registered [ 1894.529680] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 06:03:10 (1783072990) [ 1894.766627] LNet: 108301:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1894.766627] mask usage: [+|-] ... [ 1894.897778] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1894.913619] alg: No test for adler32 (adler32-zlib) [ 1895.813175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1895.852069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1895.869507] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 1896.408251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.18@tcp [ 1896.471794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg318-client@tcp [ 1901.049908] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 06:03:16 (1783072996) [ 1901.174627] LNet: 109031:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1901.188148] LNetError: 109031:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1901.215700] LNet: Removed LNI 192.168.203.18@tcp [ 1901.644770] Key type .llcrypt unregistered [ 1901.649050] Key type ._llcrypt unregistered [ 1905.915403] Key type ._llcrypt registered [ 1905.920922] Key type .llcrypt registered [ 1906.074888] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1906.091718] alg: No test for adler32 (adler32-zlib) [ 1906.935582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 1906.953947] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 1906.978470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1911.250822] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 06:03:27 (1783073007) [ 1912.313768] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 1913.398548] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 06:03:29 (1783073009) [ 1914.372592] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 1915.530776] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 06:03:31 (1783073011) [ 1915.597591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1915.616719] LNet: 110446:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1915.627387] LNetError: 110446:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1915.648234] LNet: Removed LNI 192.168.203.18@tcp [ 1915.677392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1915.723464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1915.743622] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 1916.315797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1916.348702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1920.670686] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 06:03:36 (1783073016) [ 1920.749174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1920.771474] LNet: 111156:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1920.780242] LNetError: 111156:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1920.805384] LNet: Removed LNI 192.168.203.18@tcp [ 1920.818968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1920.871757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 1920.894693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1921.321447] Key type .llcrypt unregistered [ 1921.324507] Key type ._llcrypt unregistered [ 1928.929900] Key type ._llcrypt registered [ 1928.932573] Key type .llcrypt registered [ 1929.899284] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 06:03:45 (1783073025) [ 1930.034501] LNet: 111965:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1930.034501] mask usage: [+|-] ... [ 1930.151705] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1930.163317] alg: No test for adler32 (adler32-zlib) [ 1931.045918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1931.091449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1931.109727] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 1931.639676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1931.700378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.18@tcp [ 1936.033158] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 06:03:51 (1783073031) [ 1936.208187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1936.224520] LNet: 112716:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1936.230060] LNetError: 112716:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1936.246430] LNet: Removed LNI 192.168.203.18@tcp [ 1936.269467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1936.310509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 1936.335816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 1940.872401] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 06:03:56 (1783073036) [ 1941.095416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1941.114694] LNet: 113296:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1941.133900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1941.208626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1941.551812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1941.591712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1941.652537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 1941.966918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1942.007367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1942.074487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1942.099373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1942.148408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1942.450851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1942.484193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1942.537530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 1942.883878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1942.913669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1947.686289] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 06:04:03 (1783073043) [ 1947.815799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1947.859024] LNet: 113925:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1947.866391] LNet: 113925:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1947.885590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1947.954917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1947.990038] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 1948.629511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1948.646595] LNet: Added LNI 192.168.203.18@tcp2 [8/256/0/180] [ 1949.289674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1949.339266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.18@tcp [ 1949.530918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1949.558222] LNetError: 114209:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1949.566860] LNet: Removed LNI 192.168.203.18@tcp [ 1949.591965] LNet: Removed LNI 192.168.203.18@tcp2 [ 1949.632307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1949.688407] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 1949.692710] LNet: Added LNI 192.168.203.18@tcp2 [8/256/0/180] [ 1949.696730] LNet: 114215:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1950.093559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.18@tcp [ 1950.239038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1950.252717] LNet: 114252:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1950.256794] LNet: 114252:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1951.263111] LNet: Removed LNI 192.168.203.18@tcp [ 1951.269839] LNetError: 114252:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1951.274063] LNetError: 114252:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1951.290264] LNet: Removed LNI 192.168.203.18@tcp2 [ 1951.311144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1951.352989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1951.374996] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 1951.951492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1951.980166] LNet: Added LNI 192.168.203.18@tcp2 [8/256/0/180] [ 1952.624531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.18@tcp [ 1952.663256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1952.695154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.18@tcp [ 1952.850486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1953.887303] LNet: Removed LNI 192.168.203.18@tcp [ 1953.889522] LNetError: 114536:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1953.898585] LNetError: 114536:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1953.915730] LNet: Removed LNI 192.168.203.18@tcp2 [ 1953.941431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1954.018124] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 1954.022497] LNet: Added LNI 192.168.203.18@tcp2 [8/256/0/180] [ 1954.028448] LNet: 114543:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1954.498436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.18@tcp [ 1954.709872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1954.727931] LNet: 114580:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1954.731925] LNet: 114580:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1954.740995] LNet: Removed LNI 192.168.203.18@tcp [ 1955.816779] LNet: Removed LNI 192.168.203.18@tcp2 [ 1955.855585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1955.942485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1955.970635] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 1956.584323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1956.603505] LNet: Added LNI 192.168.203.18@tcp2 [8/256/0/180] [ 1957.335810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1957.386602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.18@tcp [ 1957.560756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1958.623359] LNet: Removed LNI 192.168.203.18@tcp [ 1958.626624] LNetError: 114859:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1958.639767] LNetError: 114859:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1958.666519] LNet: Removed LNI 192.168.203.18@tcp2 [ 1958.688297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1958.744920] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 1958.753453] LNet: Added LNI 192.168.203.18@tcp2 [8/256/0/180] [ 1958.758239] LNet: 114865:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1959.123072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.18@tcp [ 1959.273838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1960.351153] LNet: Removed LNI 192.168.203.18@tcp [ 1960.363477] LNet: Removed LNI 192.168.203.18@tcp2 [ 1960.387958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1960.453504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1960.476270] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 1961.013756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1961.029552] LNet: Added LNI 192.168.203.18@tcp2 [8/256/0/180] [ 1961.633521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.18@tcp [ 1961.671756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1961.719472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.18@tcp [ 1961.893846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1962.975171] LNet: Removed LNI 192.168.203.18@tcp [ 1964.006649] LNet: Removed LNI 192.168.203.18@tcp2 [ 1964.018619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1964.075618] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 1964.087592] LNet: Added LNI 192.168.203.18@tcp2 [8/256/0/180] [ 1964.095402] LNet: 115194:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1964.535082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.18@tcp [ 1964.712370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1964.735137] LNet: 115231:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1964.740616] LNet: 115231:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1964.747575] LNet: Removed LNI 192.168.203.18@tcp [ 1965.802513] LNet: Removed LNI 192.168.203.18@tcp2 [ 1965.844575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1965.893985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1965.915860] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 1966.434419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1966.459915] LNet: Added LNI 192.168.203.18@tcp2 [8/256/0/180] [ 1967.056686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1967.086995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1967.137146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.18@tcp [ 1967.399993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1967.428237] LNetError: 115527:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1967.433694] LNetError: 115527:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1967.439039] LNet: Removed LNI 192.168.203.18@tcp [ 1967.449859] LNet: Removed LNI 192.168.203.18@tcp2 [ 1967.463329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1967.517478] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 1967.522395] LNet: Added LNI 192.168.203.18@tcp2 [8/256/0/180] [ 1967.942469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.18@tcp [ 1968.213885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1969.247239] LNet: Removed LNI 192.168.203.18@tcp [ 1969.258696] LNet: Removed LNI 192.168.203.18@tcp2 [ 1969.275204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1969.329278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1969.355753] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 1969.908688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1969.923756] LNet: Added LNI 192.168.203.18@tcp2 [8/256/0/180] [ 1970.594323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.18@tcp [ 1970.622721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1970.655128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1970.694329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.18@tcp [ 1970.962317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1971.999441] LNet: Removed LNI 192.168.203.18@tcp [ 1973.030877] LNet: Removed LNI 192.168.203.18@tcp2 [ 1973.060393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1973.103691] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 1973.113329] LNet: Added LNI 192.168.203.18@tcp2 [8/256/0/180] [ 1973.116958] LNet: 115893:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1973.120595] LNet: 115893:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1973.449733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.18@tcp [ 1978.313693] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 1979.492599] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 06:04:35 (1783073075) [ 1979.601749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1979.637530] LNet: Removed LNI 192.168.203.18@tcp [ 1980.714470] LNet: Removed LNI 192.168.203.18@tcp2 [ 1980.738561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1980.966780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1980.993287] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 1981.508400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1981.525147] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1982.154924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1982.169985] LNet: 116925:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1982.174177] LNet: 116925:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1982.180627] LNet: Removed LNI 10.1.2.3@tcp [ 1982.190044] LNet: Removed LNI 192.168.203.18@tcp [ 1982.200399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1982.246576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 1982.261027] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 1982.264480] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1982.520754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 1982.538384] LNet: Removed LNI 192.168.203.18@tcp [ 1982.548637] LNet: Removed LNI 10.1.2.3@tcp [ 1986.662863] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 06:04:42 (1783073082) [ 1986.739610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1986.766862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1986.838708] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1986.919790] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1986.923491] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1986.928121] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1989.182832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1989.197786] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 1989.781264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1989.796734] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1991.200118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.18@tcp [ 1991.227935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 1991.277041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.203.18@tcp -m PUT -e local_timeout -r 1 [ 1991.335171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 1991.390620] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1991.395912] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1991.554613] LNetError: 117591:0:(lib-move.c:4714:lnet_parse()) 192.168.203.18@tcp, src 192.168.203.18@tcp: Dropping PUT (error -108 looking up sender) [ 1992.608264] LNetError: 118050:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1992.615243] LNetError: 118050:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 9 previous similar messages [ 1992.623325] LNet: Removed LNI 10.1.2.3@tcp [ 1993.640807] LNet: Removed LNI 192.168.203.18@tcp [ 1993.914547] Key type .llcrypt unregistered [ 1993.916427] Key type ._llcrypt unregistered [ 2000.640970] Key type ._llcrypt registered [ 2000.642629] Key type .llcrypt registered [ 2001.513899] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 06:04:57 (1783073097) [ 2001.708735] LNet: 118820:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2001.708735] mask usage: [+|-] ... [ 2001.827566] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2001.844060] alg: No test for adler32 (adler32-zlib) [ 2002.713741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2002.983616] LNet: 118901:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2003.428710] Key type .llcrypt unregistered [ 2003.432291] Key type ._llcrypt unregistered [ 2007.780818] Key type ._llcrypt registered [ 2007.783622] Key type .llcrypt registered [ 2007.825632] LNet: 119129:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2007.825632] mask usage: [+|-] ... [ 2007.972772] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2007.986915] alg: No test for adler32 (adler32-zlib) [ 2008.850816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2008.883733] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 2014.671502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.118@tcp [ 2014.869962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2015.287715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.118@tcp [ 2015.466703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2015.487512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2015.515409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2015.538756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2015.563091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2015.590618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2015.620628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2015.638730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2015.650629] LNet: 119350:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2015.665691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2015.687502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2015.712144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2015.755154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2015.800851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2015.840359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2016.326375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2016.493432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2016.521965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2016.548513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2016.573904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2016.596543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2016.619258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2016.641949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2016.667922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2016.684689] LNet: 119467:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2016.699155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2016.726578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2016.756414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2016.780249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2016.802869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2016.832459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2017.274976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.118@tcp [ 2017.417055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2017.440179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2017.466936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2017.487943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2017.511317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2017.542430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2017.572870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2017.603965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2017.641886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2017.672530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2017.694138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2017.723480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2017.747369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2017.773712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2018.244570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2018.452312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2018.485362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2018.512571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2018.536531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2018.557727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2018.578785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2018.600240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2018.622551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2018.645654] LNet: 119701:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2018.651218] LNet: 119701:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 2018.670589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2018.717960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2018.741973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2018.765830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2018.796567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2018.834144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2019.075994] LNet: 119747:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2019.080978] LNetError: 119747:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2019.097398] LNet: Removed LNI 192.168.203.18@tcp [ 2019.343522] Key type .llcrypt unregistered [ 2019.345737] Key type ._llcrypt unregistered [ 2030.061732] Key type ._llcrypt registered [ 2030.064838] Key type .llcrypt registered [ 2031.020663] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 06:05:26 (1783073126) [ 2031.165735] LNet: 120552:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2031.165735] mask usage: [+|-] ... [ 2031.315799] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2031.329854] alg: No test for adler32 (adler32-zlib) [ 2032.170334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2032.411249] LNet: 120634:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2032.817759] Key type .llcrypt unregistered [ 2032.822762] Key type ._llcrypt unregistered [ 2036.118218] Key type ._llcrypt registered [ 2036.120064] Key type .llcrypt registered [ 2036.141723] LNet: 120863:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2036.141723] mask usage: [+|-] ... [ 2036.242725] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2036.255212] alg: No test for adler32 (adler32-zlib) [ 2037.105891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2037.123267] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 2041.829745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.118@tcp [ 2047.095573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2047.116556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2047.132820] LNet: Added LNI 192.168.203.18@tcp1 [8/256/0/180] [ 2047.747159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2048.086585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.118@tcp [ 2048.225791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2048.245910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2048.276154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2048.299907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2048.326898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2048.352708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2048.383338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2048.403615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2048.415240] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.118@tcp [ 2048.418532] LNet: 121285:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2048.431527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2048.454424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2048.479800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2048.504398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2048.526131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2048.549381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2048.937983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2049.081773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2049.100608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2049.121375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2049.140263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2049.162301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2049.185436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2049.204094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2049.231769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2049.249481] LNet: 121402:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2049.270048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2049.289476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2049.311598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2049.331078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2049.357527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2049.378496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2049.767229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.118@tcp [ 2049.910551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2049.938520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2049.963801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2049.987983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2050.008859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2050.028244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2050.047455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2050.072801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2050.094329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2050.119858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2050.141879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2050.161766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2050.180180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2050.203298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2050.562926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2050.687528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2050.707300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2050.727385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2050.756595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2050.778181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2050.805538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2050.826900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2050.846816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2050.859700] LNet: 121637:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2050.864987] LNet: 121637:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 2050.875877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2050.897236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2050.927635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2050.950893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2050.972397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2050.992839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2054.039997] LNet: 121717:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2054.044566] LNetError: 121717:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2054.049554] LNet: Removed LNI 192.168.203.18@tcp [ 2054.056480] LNet: Removed LNI 192.168.203.18@tcp1 [ 2054.281705] Key type .llcrypt unregistered [ 2054.286450] Key type ._llcrypt unregistered [ 2063.431322] Key type ._llcrypt registered [ 2063.433080] Key type .llcrypt registered [ 2064.273764] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 06:06:00 (1783073160) [ 2068.505389] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 06:06:04 (1783073164) [ 2072.713420] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 06:06:08 (1783073168) [ 2077.365609] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 06:06:13 (1783073173) [ 2081.797463] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 06:06:17 (1783073177) [ 2086.256615] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 06:06:22 (1783073182) [ 2090.821269] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 06:06:26 (1783073186) [ 2095.269881] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 06:06:31 (1783073191) [ 2099.804827] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 06:06:35 (1783073195) [ 2103.710981] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 06:06:39 (1783073199) [ 2107.997590] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 06:06:43 (1783073203) [ 2112.746144] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 06:06:48 (1783073208) [ 2116.653341] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 06:06:52 (1783073212) [ 2120.361829] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 06:06:56 (1783073216) [ 2124.544388] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 06:07:00 (1783073220) [ 2128.676971] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 06:07:04 (1783073224) [ 2133.290318] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 06:07:09 (1783073229) [ 2137.530920] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 06:07:13 (1783073233) [ 2137.669697] LNet: 132251:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2137.669697] mask usage: [+|-] ... [ 2137.773304] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2137.786643] alg: No test for adler32 (adler32-zlib) [ 2138.660873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2138.756541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2138.776898] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 2139.500707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2139.518763] LNet: 132422:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2139.524825] LNetError: 132422:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2139.541926] LNet: Removed LNI 192.168.203.18@tcp [ 2139.555913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2139.599863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2139.620710] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 2143.769204] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 06:07:19 (1783073239) [ 2148.519016] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 06:07:24 (1783073244) [ 2152.781079] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 06:07:28 (1783073248) [ 2157.143824] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 06:07:33 (1783073253) [ 2161.578287] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 06:07:37 (1783073257) [ 2166.072053] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 06:07:41 (1783073261) [ 2170.449808] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 06:07:46 (1783073266) [ 2174.594393] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 06:07:50 (1783073270) [ 2178.438933] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 06:07:54 (1783073274) [ 2182.814149] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 06:07:58 (1783073278) [ 2186.851850] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 06:08:02 (1783073282) [ 2190.871853] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 06:08:06 (1783073286) [ 2195.121556] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 1936 sec ========== 06:08:10 (1783073290) [ 2195.309825] LNet: 139880:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2195.314789] LNetError: 139880:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2195.325642] LNet: Removed LNI 192.168.203.18@tcp [ 2195.731268] Key type .llcrypt unregistered [ 2195.739297] Key type ._llcrypt unregistered [ 2202.928225] Key type ._llcrypt registered [ 2202.931556] Key type .llcrypt registered [ 2202.983566] LNet: 140336:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2202.983566] mask usage: [+|-] ... [ 2203.224713] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2203.237141] alg: No test for adler32 (adler32-zlib) [ 2204.184168] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 2204.334390] LNet: Added LNI 192.168.203.18@tcp [8/256/0/180] [ 2205.959559] Key type lgssc registered [ 2206.681303] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2245.168144] Lustre: Mounted lustre-client [ 2245.253771] LNet: 141819:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2245.253771] mask usage: [+|-] ... [ 2248.739935] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2253.550671] Lustre: Unmounted lustre-client [ 2285.304416] Key type lgssc unregistered [ 2285.507349] LNet: 142801:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2285.514302] LNetError: 142801:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2285.531050] LNet: Removed LNI 192.168.203.18@tcp [ 2285.992084] Key type .llcrypt unregistered [ 2285.998026] Key type ._llcrypt unregistered